2021-05-04T00:00:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:03.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:07.027Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.027Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.033Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.033Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=213d8658-7efd-4655-8b9d-ff527c2223c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.033Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=213d8658-7efd-4655-8b9d-ff527c2223c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2183541a 2021-05-04T00:00:07.033Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=213d8658-7efd-4655-8b9d-ff527c2223c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.033Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=213d8658-7efd-4655-8b9d-ff527c2223c8 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.044Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=213d8658-7efd-4655-8b9d-ff527c2223c8 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:00:07.044Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=213d8658-7efd-4655-8b9d-ff527c2223c8 ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:00:07.047Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.049Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38a71b6a-0765-4c0f-9c14-74af5576065d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38a71b6a-0765-4c0f-9c14-74af5576065d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b284cd5 2021-05-04T00:00:07.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38a71b6a-0765-4c0f-9c14-74af5576065d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38a71b6a-0765-4c0f-9c14-74af5576065d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.063Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.065Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a4795dd-462a-432a-aca7-03b635d6113a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a4795dd-462a-432a-aca7-03b635d6113a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7af03da6 2021-05-04T00:00:07.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a4795dd-462a-432a-aca7-03b635d6113a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a4795dd-462a-432a-aca7-03b635d6113a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.070Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.073Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.074Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17782c9d-be83-46fb-b4de-15d39c0e6819 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.074Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17782c9d-be83-46fb-b4de-15d39c0e6819 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29d097bf 2021-05-04T00:00:07.074Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17782c9d-be83-46fb-b4de-15d39c0e6819 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.074Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17782c9d-be83-46fb-b4de-15d39c0e6819 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.078Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.080Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.080Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf612f35-a4db-42ab-8d38-5b137df334f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.080Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf612f35-a4db-42ab-8d38-5b137df334f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12c8251b 2021-05-04T00:00:07.080Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf612f35-a4db-42ab-8d38-5b137df334f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.080Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf612f35-a4db-42ab-8d38-5b137df334f6 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.085Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.087Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.087Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2b4f131-e57f-4bbb-b135-c35dcc785987 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.087Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2b4f131-e57f-4bbb-b135-c35dcc785987 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2bbf9d73 2021-05-04T00:00:07.087Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2b4f131-e57f-4bbb-b135-c35dcc785987 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.087Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2b4f131-e57f-4bbb-b135-c35dcc785987 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.091Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.093Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.093Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1e0371e-1314-4e32-9ce7-ee1e4e1e5ee5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.093Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1e0371e-1314-4e32-9ce7-ee1e4e1e5ee5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@505998af 2021-05-04T00:00:07.093Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1e0371e-1314-4e32-9ce7-ee1e4e1e5ee5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.093Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1e0371e-1314-4e32-9ce7-ee1e4e1e5ee5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.097Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.099Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003406bd-f476-4a32-bffb-63d17efff1f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003406bd-f476-4a32-bffb-63d17efff1f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7d199074 2021-05-04T00:00:07.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003406bd-f476-4a32-bffb-63d17efff1f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003406bd-f476-4a32-bffb-63d17efff1f5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.103Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.105Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e7f01be-dacd-4c46-bede-aa3b8b867eba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e7f01be-dacd-4c46-bede-aa3b8b867eba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e2a8231 2021-05-04T00:00:07.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e7f01be-dacd-4c46-bede-aa3b8b867eba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e7f01be-dacd-4c46-bede-aa3b8b867eba PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.110Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.112Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76b1a1b0-1981-4654-b61a-845aa5651d52 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76b1a1b0-1981-4654-b61a-845aa5651d52 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7929c682 2021-05-04T00:00:07.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76b1a1b0-1981-4654-b61a-845aa5651d52 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76b1a1b0-1981-4654-b61a-845aa5651d52 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.116Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.117Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.118Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8076f3c2-3cdf-4a76-a096-c972c88fcfed PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.118Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8076f3c2-3cdf-4a76-a096-c972c88fcfed PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6529a5e8 2021-05-04T00:00:07.118Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8076f3c2-3cdf-4a76-a096-c972c88fcfed PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.118Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8076f3c2-3cdf-4a76-a096-c972c88fcfed PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.121Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.123Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15befe02-f5a0-4203-ad95-2f18dc0ec4c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15befe02-f5a0-4203-ad95-2f18dc0ec4c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54020309 2021-05-04T00:00:07.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15befe02-f5a0-4203-ad95-2f18dc0ec4c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15befe02-f5a0-4203-ad95-2f18dc0ec4c1 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.127Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.128Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bffae09-a016-4dd8-b7d5-cb58428c4319 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bffae09-a016-4dd8-b7d5-cb58428c4319 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ec328b2 2021-05-04T00:00:07.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bffae09-a016-4dd8-b7d5-cb58428c4319 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bffae09-a016-4dd8-b7d5-cb58428c4319 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.132Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.133Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.133Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96709df1-5fb4-4ecb-91e5-4c8d9d338d01 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.133Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96709df1-5fb4-4ecb-91e5-4c8d9d338d01 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1969e63e 2021-05-04T00:00:07.133Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96709df1-5fb4-4ecb-91e5-4c8d9d338d01 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.133Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96709df1-5fb4-4ecb-91e5-4c8d9d338d01 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.137Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.138Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.139Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fadbeb6c-f927-4a90-ab01-caec33825c03 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.139Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fadbeb6c-f927-4a90-ab01-caec33825c03 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3147161e 2021-05-04T00:00:07.139Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fadbeb6c-f927-4a90-ab01-caec33825c03 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.139Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fadbeb6c-f927-4a90-ab01-caec33825c03 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.142Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.144Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.144Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=587d623c-a491-494c-801b-c83324945c8c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.144Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=587d623c-a491-494c-801b-c83324945c8c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75995174 2021-05-04T00:00:07.144Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=587d623c-a491-494c-801b-c83324945c8c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.144Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=587d623c-a491-494c-801b-c83324945c8c PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.148Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.149Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ae99eb2-12a9-46dd-a627-13636e9704a7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ae99eb2-12a9-46dd-a627-13636e9704a7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@422861cd 2021-05-04T00:00:07.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ae99eb2-12a9-46dd-a627-13636e9704a7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ae99eb2-12a9-46dd-a627-13636e9704a7 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.153Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.155Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75b47233-ad23-4f07-8258-63096aa2ed13 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75b47233-ad23-4f07-8258-63096aa2ed13 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14a627a0 2021-05-04T00:00:07.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75b47233-ad23-4f07-8258-63096aa2ed13 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75b47233-ad23-4f07-8258-63096aa2ed13 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.159Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.160Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f523687c-2985-4619-98d8-9472e7db60f2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f523687c-2985-4619-98d8-9472e7db60f2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1dd824cc 2021-05-04T00:00:07.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f523687c-2985-4619-98d8-9472e7db60f2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f523687c-2985-4619-98d8-9472e7db60f2 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:07.662Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:07.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=292a250b-6eee-4db9-93ae-fae5d0953f19 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:07.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=292a250b-6eee-4db9-93ae-fae5d0953f19 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2275c4f2 2021-05-04T00:00:07.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=292a250b-6eee-4db9-93ae-fae5d0953f19 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:07.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=292a250b-6eee-4db9-93ae-fae5d0953f19 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:08.052Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ug3ej309f6hnlqegc4h6u87i158 2021-05-04T00:00:08.053Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01nmoy1box25i1lqcjys2jjy9w121 2021-05-04T00:00:08.053Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01clnjo0wy7uxl1l8920bjkbed4154 2021-05-04T00:00:08.053Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0191c8lw3pgkax14u3uxafsgryg135 2021-05-04T00:00:08.053Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01pkqbi5uwigh0t21zmzzeixe8119 2021-05-04T00:00:08.053Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01xqkex8mgddlijb4dmjxljl63167 2021-05-04T00:00:08.053Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01qr4d7pyaargr1f5z0thlwnkzk123 2021-05-04T00:00:08.053Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node014tencdboabvrxsebzgrm28kv148 2021-05-04T00:00:08.053Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0shw47gi5961ic3ap9nvt12ud140 2021-05-04T00:00:08.053Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01fa4ll70kn5di3iwx5sl2cc8c120 2021-05-04T00:00:08.054Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0q2kr5zjszeev1pbw0m4qsn0l3136 2021-05-04T00:00:08.054Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01xqis9dwb9gpu15l4kywq9o8et165 2021-05-04T00:00:08.054Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0x5yzjw0am3f619jjo6pillfsh162 2021-05-04T00:00:08.054Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0sfgagty4lmgxq5hec5oyta0m144 2021-05-04T00:00:08.054Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0hmm9ed0qv3uu1akm9k8vvqune138 2021-05-04T00:00:08.054Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01eeu8tdbovwhh2j4r2uh5pcpj149 2021-05-04T00:00:08.054Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0lc104opgn90a1tbqhfjjnlcmt151 2021-05-04T00:00:08.054Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node016jeiilsb6irp1ka1j8207uohd127 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01h7da79cuy8o6oamg4rkhsr8q122 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node09aqkx0hn95on1asy8uajdpo5w143 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0x0ymoa71cr381o2a2atnot2qp157 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node09h6v4w915m2lvigojfc9wysm147 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01d339bfxzo9r31wtshi5aits95141 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node012ebdwsm7jopfwkap3x6hr5u9132 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01wp3u8yraxyuj19orh6lqyzfgw153 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01q7kk2xuic04v1x3fsvwj4peql124 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ug3c1wc3zkqn1nu9wkm4smfca161 2021-05-04T00:00:08.055Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ih8qscucs6jwgrboomvsop3t125 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01792ov5tsj0p8120cxcqd0tl1145 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01uhqsm382hdas1oj1vr7tshv7n156 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01mm3da3taayafwb4j7z82vl65131 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node079p57beybtw1nfyhkh4d3voh159 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0lnc0uugxnu0pqglzq2w6mz8q130 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node090f0n1zwn9017nid0ghp1wfm155 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0mdtq0xit4tvxsrn1m1u6wuaa163 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0yvek9agqcqnce3xrh7n77n38166 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01893l62c3z1me1mf43w0l4hgxh126 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0yoa8cq648lchr7erxpw67m9s129 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ttga55oq73hzc8t1yref2lwm150 2021-05-04T00:00:08.056Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node05ji9malrggwgtye8bhlntlry152 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ilaxohlmnqh81fz2wopaby2s2168 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0xc9weoxniqacfizdxhtxumip128 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01cvojz5gu9o3dwt534bvg6krj164 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node04i8dwltihqfzfvwde4ctnsqr142 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0gmz4rtnqe1pr1h8jx3ccu7zcw146 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ccrxq5hs3aa3gv10qu3ulu99137 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0wgob47w0fi911mt72g70uydyb139 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01f7bevt0ltlg818p20je51g9nz160 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01v9l3pi0mdzr2zo0c8mpayouo133 2021-05-04T00:00:08.057Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01xa15ue22auao3uxm4pvepte6134 2021-05-04T00:00:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:10.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:10.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:10.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8a4fde7-4479-4bed-8d5c-455ff2503ca5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:10.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8a4fde7-4479-4bed-8d5c-455ff2503ca5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2352ff78 2021-05-04T00:00:10.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8a4fde7-4479-4bed-8d5c-455ff2503ca5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:10.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8a4fde7-4479-4bed-8d5c-455ff2503ca5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:13.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:13.665Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:13.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bc39cb1-110a-4952-aedf-dc949bb4aace PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:13.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bc39cb1-110a-4952-aedf-dc949bb4aace PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66c5795 2021-05-04T00:00:13.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bc39cb1-110a-4952-aedf-dc949bb4aace PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:13.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bc39cb1-110a-4952-aedf-dc949bb4aace PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:15.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:16.440Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T00:00:16.441Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 64000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:16.662Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:16.664Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=493c3b46-4848-485e-9b28-72c7575bab0a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=493c3b46-4848-485e-9b28-72c7575bab0a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a4eedd0 2021-05-04T00:00:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=493c3b46-4848-485e-9b28-72c7575bab0a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=493c3b46-4848-485e-9b28-72c7575bab0a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:19.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:19.666Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbfeb13b-db06-4823-8cd0-45cfb30b4162 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbfeb13b-db06-4823-8cd0-45cfb30b4162 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5610f03f 2021-05-04T00:00:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbfeb13b-db06-4823-8cd0-45cfb30b4162 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbfeb13b-db06-4823-8cd0-45cfb30b4162 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:22.662Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:22.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ce7444-ced9-43fb-90a0-6eee962b4d31 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ce7444-ced9-43fb-90a0-6eee962b4d31 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@628d7cba 2021-05-04T00:00:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ce7444-ced9-43fb-90a0-6eee962b4d31 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ce7444-ced9-43fb-90a0-6eee962b4d31 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:25.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:25.666Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:25.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53259362-365f-4f68-bb89-82df8d50f5fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:25.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53259362-365f-4f68-bb89-82df8d50f5fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6394f81 2021-05-04T00:00:25.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53259362-365f-4f68-bb89-82df8d50f5fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:25.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53259362-365f-4f68-bb89-82df8d50f5fd PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:00:28.662Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:28.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:28.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=483fcafd-a71c-4aa7-9b89-7470b4ccfeef PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:28.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=483fcafd-a71c-4aa7-9b89-7470b4ccfeef PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2bb412fb 2021-05-04T00:00:28.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=483fcafd-a71c-4aa7-9b89-7470b4ccfeef PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:28.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=483fcafd-a71c-4aa7-9b89-7470b4ccfeef PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:29.209Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:00:31.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:31.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:31.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75973b76-41b3-4766-93a8-903cd7786bb5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:31.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75973b76-41b3-4766-93a8-903cd7786bb5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a748eba 2021-05-04T00:00:31.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75973b76-41b3-4766-93a8-903cd7786bb5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:31.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75973b76-41b3-4766-93a8-903cd7786bb5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:31.668Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=75973b76-41b3-4766-93a8-903cd7786bb5 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:00:31.668Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=75973b76-41b3-4766-93a8-903cd7786bb5 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:00:34.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:34.702Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:34.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c671cebb-688d-4a68-b07c-a22b1cf4bff8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:34.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c671cebb-688d-4a68-b07c-a22b1cf4bff8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@739545e2 2021-05-04T00:00:34.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c671cebb-688d-4a68-b07c-a22b1cf4bff8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:34.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c671cebb-688d-4a68-b07c-a22b1cf4bff8 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:00:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:00:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:00:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:00:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:00:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:00:37.116Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:00:37.116Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:00:37.118Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:00:37.118Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:00:37.119Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:00:37.120Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:00:37.120Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:00:37.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:37.665Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:37.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26586c99-54af-44de-af1b-1dff143f21ce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:37.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26586c99-54af-44de-af1b-1dff143f21ce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f9c5956 2021-05-04T00:00:37.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26586c99-54af-44de-af1b-1dff143f21ce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:37.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26586c99-54af-44de-af1b-1dff143f21ce PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:40.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:40.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:40.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6cc9e0aa-223f-4cfc-a03f-d4cf8bda92d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:40.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6cc9e0aa-223f-4cfc-a03f-d4cf8bda92d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3434a794 2021-05-04T00:00:40.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6cc9e0aa-223f-4cfc-a03f-d4cf8bda92d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:40.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6cc9e0aa-223f-4cfc-a03f-d4cf8bda92d9 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:43.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:43.700Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:43.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e3d3714-1502-4fa4-a035-f2292a13a7c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:43.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e3d3714-1502-4fa4-a035-f2292a13a7c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6aebfef6 2021-05-04T00:00:43.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e3d3714-1502-4fa4-a035-f2292a13a7c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:43.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e3d3714-1502-4fa4-a035-f2292a13a7c8 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:46.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:46.666Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:46.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=524a478b-ed91-42ff-a884-f06c8d3883ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:46.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=524a478b-ed91-42ff-a884-f06c8d3883ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33080c1c 2021-05-04T00:00:46.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=524a478b-ed91-42ff-a884-f06c8d3883ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:46.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=524a478b-ed91-42ff-a884-f06c8d3883ff PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:49.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:49.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:49.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffe6263e-f81c-4ed3-9707-132e41250bc3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:49.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffe6263e-f81c-4ed3-9707-132e41250bc3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e03a5c2 2021-05-04T00:00:49.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffe6263e-f81c-4ed3-9707-132e41250bc3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:49.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffe6263e-f81c-4ed3-9707-132e41250bc3 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:52.700Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:52.703Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:52.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8359c6-7d1d-47d3-a89b-3d4ebabeb5c0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:52.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8359c6-7d1d-47d3-a89b-3d4ebabeb5c0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f9ab791 2021-05-04T00:00:52.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8359c6-7d1d-47d3-a89b-3d4ebabeb5c0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:52.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8359c6-7d1d-47d3-a89b-3d4ebabeb5c0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:55.701Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:55.704Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:55.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb4a2f47-e647-46f2-9b9d-668659a0e06e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:55.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb4a2f47-e647-46f2-9b9d-668659a0e06e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a33eada 2021-05-04T00:00:55.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb4a2f47-e647-46f2-9b9d-668659a0e06e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:55.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb4a2f47-e647-46f2-9b9d-668659a0e06e PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:00:58.662Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:58.665Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:00:58.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2368c9b9-d969-4ff9-a580-e148bb9d078d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:00:58.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2368c9b9-d969-4ff9-a580-e148bb9d078d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39a8776c 2021-05-04T00:00:58.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2368c9b9-d969-4ff9-a580-e148bb9d078d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:00:58.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2368c9b9-d969-4ff9-a580-e148bb9d078d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:01.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:01.703Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:01.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac896de-e525-4711-a2d7-f99a79b0fe62 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:01.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac896de-e525-4711-a2d7-f99a79b0fe62 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a76e82e 2021-05-04T00:01:01.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac896de-e525-4711-a2d7-f99a79b0fe62 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:01.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac896de-e525-4711-a2d7-f99a79b0fe62 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:04.701Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:04.703Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:04.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83776311-822d-4bb2-9d74-a46aba4a0e2d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:04.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83776311-822d-4bb2-9d74-a46aba4a0e2d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60fae8ce 2021-05-04T00:01:04.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83776311-822d-4bb2-9d74-a46aba4a0e2d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:04.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83776311-822d-4bb2-9d74-a46aba4a0e2d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:07.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:07.665Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:07.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08916ef8-15a0-4850-be09-5bd25dc64beb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:07.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08916ef8-15a0-4850-be09-5bd25dc64beb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35cabb31 2021-05-04T00:01:07.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08916ef8-15a0-4850-be09-5bd25dc64beb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:07.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08916ef8-15a0-4850-be09-5bd25dc64beb PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:10.662Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:10.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:10.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=847d91b4-a58e-4f17-b62a-70dc05a4b235 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:10.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=847d91b4-a58e-4f17-b62a-70dc05a4b235 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4fa3b9f3 2021-05-04T00:01:10.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=847d91b4-a58e-4f17-b62a-70dc05a4b235 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:10.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=847d91b4-a58e-4f17-b62a-70dc05a4b235 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:13.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:13.703Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37f3e6f1-8de9-4c37-a453-387542d4934e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37f3e6f1-8de9-4c37-a453-387542d4934e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32e91fb1 2021-05-04T00:01:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37f3e6f1-8de9-4c37-a453-387542d4934e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37f3e6f1-8de9-4c37-a453-387542d4934e PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:16.702Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:16.709Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:16.710Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a3685d9-8396-44c2-bb3c-1cc23c388266 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:16.710Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a3685d9-8396-44c2-bb3c-1cc23c388266 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e23dba 2021-05-04T00:01:16.710Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a3685d9-8396-44c2-bb3c-1cc23c388266 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:16.710Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a3685d9-8396-44c2-bb3c-1cc23c388266 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:19.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:19.666Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=494690ca-9c22-4f17-a5a7-ef7e186a080b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=494690ca-9c22-4f17-a5a7-ef7e186a080b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65910d04 2021-05-04T00:01:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=494690ca-9c22-4f17-a5a7-ef7e186a080b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=494690ca-9c22-4f17-a5a7-ef7e186a080b PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:19.983Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:01:19.983Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:19.987Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:19.987Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:19.988Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:19.988Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:19.989Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:19.993Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:19.994Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:19.994Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:19.996Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:19.996Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:19.997Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:19.997Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:19.999Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.000Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.001Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.001Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.003Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.003Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.004Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.004Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.006Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.007Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.008Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.008Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.010Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.010Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.011Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.011Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.013Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.014Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.015Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.015Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.016Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.016Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.017Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.017Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.019Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.020Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.021Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.021Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.022Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.022Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.024Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.024Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.025Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.026Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.028Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.028Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.029Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.029Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.030Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.030Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.031Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.032Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.034Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.034Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.035Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.035Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.036Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.036Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.038Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.039Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.040Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.040Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.041Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.041Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.043Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.043Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.044Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.045Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.046Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.046Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.047Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.047Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.049Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.049Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.051Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.052Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.053Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.053Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.054Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.054Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.056Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.056Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.057Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.058Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.059Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.059Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.061Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.061Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.062Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.062Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.063Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.170Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.170Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T00:01:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:20.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:20.498Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T00:01:20.500Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 128000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:22.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:22.702Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd9b0d5-b60d-4b53-9dd9-ea1d5359a1e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd9b0d5-b60d-4b53-9dd9-ea1d5359a1e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37af5c64 2021-05-04T00:01:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd9b0d5-b60d-4b53-9dd9-ea1d5359a1e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd9b0d5-b60d-4b53-9dd9-ea1d5359a1e0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:01:25.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:25.666Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:25.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=451cab10-b4d3-4fc9-bff5-571fe3d5d226 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:25.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=451cab10-b4d3-4fc9-bff5-571fe3d5d226 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4dc1c9b7 2021-05-04T00:01:25.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=451cab10-b4d3-4fc9-bff5-571fe3d5d226 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:25.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=451cab10-b4d3-4fc9-bff5-571fe3d5d226 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:25.671Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=451cab10-b4d3-4fc9-bff5-571fe3d5d226 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:01:25.671Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=451cab10-b4d3-4fc9-bff5-571fe3d5d226 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:01:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:28.665Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:28.666Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:28.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b7b90c8-4e1b-4d9b-a1fc-356a58ebf71a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:28.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b7b90c8-4e1b-4d9b-a1fc-356a58ebf71a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@595a3258 2021-05-04T00:01:28.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b7b90c8-4e1b-4d9b-a1fc-356a58ebf71a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:28.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b7b90c8-4e1b-4d9b-a1fc-356a58ebf71a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:31.666Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:31.668Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:31.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc4579c6-6aef-4a58-a960-3d3217a0d885 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:31.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc4579c6-6aef-4a58-a960-3d3217a0d885 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34ceed5a 2021-05-04T00:01:31.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc4579c6-6aef-4a58-a960-3d3217a0d885 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:31.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc4579c6-6aef-4a58-a960-3d3217a0d885 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:34.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:34.702Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:34.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4739e0c2-e891-41eb-9c4d-5d7ae2d921c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:34.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4739e0c2-e891-41eb-9c4d-5d7ae2d921c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@794fdbdb 2021-05-04T00:01:34.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4739e0c2-e891-41eb-9c4d-5d7ae2d921c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:34.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4739e0c2-e891-41eb-9c4d-5d7ae2d921c8 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:35.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:35.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:35.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:35.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:35.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:35.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:35.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:35.189Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:36.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:36.940Z [cluster2-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:36.940Z [cluster2-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:01:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:01:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:01:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:01:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:37.116Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:01:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:01:37.661Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:37.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:37.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8a836f7-79e8-452f-96ce-3cdf42ce795f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:37.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8a836f7-79e8-452f-96ce-3cdf42ce795f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a003934 2021-05-04T00:01:37.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8a836f7-79e8-452f-96ce-3cdf42ce795f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:37.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8a836f7-79e8-452f-96ce-3cdf42ce795f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:39.203Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:40.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:40.666Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:40.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ff60521-1858-45f4-90a6-8f067b763e9f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:40.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ff60521-1858-45f4-90a6-8f067b763e9f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@324004de 2021-05-04T00:01:40.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ff60521-1858-45f4-90a6-8f067b763e9f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:40.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ff60521-1858-45f4-90a6-8f067b763e9f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:43.701Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:43.704Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:43.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c8eab4f-2017-4aca-8c03-b625f8e97de9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:43.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c8eab4f-2017-4aca-8c03-b625f8e97de9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71a78bf1 2021-05-04T00:01:43.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c8eab4f-2017-4aca-8c03-b625f8e97de9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:43.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c8eab4f-2017-4aca-8c03-b625f8e97de9 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:45.208Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:46.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:46.665Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:46.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a065faa5-c91f-4383-b1de-ffe47604005c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:46.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a065faa5-c91f-4383-b1de-ffe47604005c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76b1071d 2021-05-04T00:01:46.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a065faa5-c91f-4383-b1de-ffe47604005c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:46.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a065faa5-c91f-4383-b1de-ffe47604005c PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:47.562Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:01:49.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:49.664Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:49.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a655bfa-2334-4871-b3da-e5cf8cb2cb9b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:49.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a655bfa-2334-4871-b3da-e5cf8cb2cb9b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d435966 2021-05-04T00:01:49.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a655bfa-2334-4871-b3da-e5cf8cb2cb9b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:49.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a655bfa-2334-4871-b3da-e5cf8cb2cb9b PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:49.668Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9a655bfa-2334-4871-b3da-e5cf8cb2cb9b ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:01:49.669Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=9a655bfa-2334-4871-b3da-e5cf8cb2cb9b ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:01:50.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:50.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:50.183Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:01:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:50.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:50.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:50.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:01:50.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:01:50.188Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:01:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:01:52.665Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:52.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:52.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=914fb745-1d9a-4c3a-ac41-4d127d07b8c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:52.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=914fb745-1d9a-4c3a-ac41-4d127d07b8c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f97c2e4 2021-05-04T00:01:52.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=914fb745-1d9a-4c3a-ac41-4d127d07b8c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:52.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=914fb745-1d9a-4c3a-ac41-4d127d07b8c2 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:52.705Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=914fb745-1d9a-4c3a-ac41-4d127d07b8c2 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:01:52.705Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=914fb745-1d9a-4c3a-ac41-4d127d07b8c2 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:01:54.210Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:55.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:55.702Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:55.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16e5b82e-26ad-4a51-9fa1-e4dca020a0a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:55.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16e5b82e-26ad-4a51-9fa1-e4dca020a0a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32a226e4 2021-05-04T00:01:55.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16e5b82e-26ad-4a51-9fa1-e4dca020a0a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:55.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16e5b82e-26ad-4a51-9fa1-e4dca020a0a0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:01:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:01:58.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:58.665Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:01:58.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32c96158-8c32-4dc6-81da-db4b845d75cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:01:58.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32c96158-8c32-4dc6-81da-db4b845d75cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4cd54642 2021-05-04T00:01:58.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32c96158-8c32-4dc6-81da-db4b845d75cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:01:58.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32c96158-8c32-4dc6-81da-db4b845d75cf PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:02:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:01.702Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:01.705Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:01.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cc5229b-1462-445b-a042-32212b19f79c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:02:01.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cc5229b-1462-445b-a042-32212b19f79c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b0abed7 2021-05-04T00:02:01.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cc5229b-1462-445b-a042-32212b19f79c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:02:01.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cc5229b-1462-445b-a042-32212b19f79c PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:02:02.215Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:02:04.701Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:05.000Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:05.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=721d44b4-420b-4fd3-9f6c-925e34c725db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:02:05.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=721d44b4-420b-4fd3-9f6c-925e34c725db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5acbcfb0 2021-05-04T00:02:05.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=721d44b4-420b-4fd3-9f6c-925e34c725db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:02:05.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=721d44b4-420b-4fd3-9f6c-925e34c725db PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:02:05.007Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=721d44b4-420b-4fd3-9f6c-925e34c725db ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:02:05.007Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=721d44b4-420b-4fd3-9f6c-925e34c725db ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:02:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:02:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:02:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:02:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:02:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:02:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:02:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:02:05.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:02:05.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:02:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:02:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:02:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:02:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:02:05.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:02:05.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:02:05.192Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:02:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:02:07.661Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:15.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:02:20.760Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:20.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1245ce02-4243-4c40-b7ba-3717f17cc71d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:02:20.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1245ce02-4243-4c40-b7ba-3717f17cc71d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@797d470b 2021-05-04T00:02:20.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1245ce02-4243-4c40-b7ba-3717f17cc71d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:02:20.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1245ce02-4243-4c40-b7ba-3717f17cc71d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:02:20.769Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1245ce02-4243-4c40-b7ba-3717f17cc71d ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:02:20.769Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=1245ce02-4243-4c40-b7ba-3717f17cc71d ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:02:20.771Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:27.763Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:27.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90577da4-6312-4779-95bd-a262a23cefd1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:02:27.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90577da4-6312-4779-95bd-a262a23cefd1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ffce930 2021-05-04T00:02:27.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90577da4-6312-4779-95bd-a262a23cefd1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:02:27.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90577da4-6312-4779-95bd-a262a23cefd1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:02:27.770Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:27.801Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:02:34.499Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:34.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2d3df1-b103-44e1-9b53-bea28a8b79ce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:02:34.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2d3df1-b103-44e1-9b53-bea28a8b79ce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51ce001e 2021-05-04T00:02:34.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2d3df1-b103-44e1-9b53-bea28a8b79ce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:02:34.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2d3df1-b103-44e1-9b53-bea28a8b79ce PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:02:34.507Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:36.101Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:36.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac2c80ca-5731-44b0-998e-c8b1c6be42dd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:02:36.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac2c80ca-5731-44b0-998e-c8b1c6be42dd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55fcf3d8 2021-05-04T00:02:36.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac2c80ca-5731-44b0-998e-c8b1c6be42dd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:02:36.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac2c80ca-5731-44b0-998e-c8b1c6be42dd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:02:36.106Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ac2c80ca-5731-44b0-998e-c8b1c6be42dd ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:02:36.106Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ac2c80ca-5731-44b0-998e-c8b1c6be42dd ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:02:36.108Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:02:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:02:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:02:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:02:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:02:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:02:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:02:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:02:37.116Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:02:37.116Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:02:37.117Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:02:37.118Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:02:37.118Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:02:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:02:42.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:42.603Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:42.609Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3240875b-bba7-4cfa-a4e0-367405716962 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:02:42.609Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3240875b-bba7-4cfa-a4e0-367405716962 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46aab373 2021-05-04T00:02:42.609Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3240875b-bba7-4cfa-a4e0-367405716962 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:02:42.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3240875b-bba7-4cfa-a4e0-367405716962 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:02:42.625Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:02:45.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:54.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:02:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:01.301Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:01.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e19243f-c0d6-403f-bd7d-6ddf1f9d2e3f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:03:01.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e19243f-c0d6-403f-bd7d-6ddf1f9d2e3f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e6777ff 2021-05-04T00:03:01.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e19243f-c0d6-403f-bd7d-6ddf1f9d2e3f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:03:01.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e19243f-c0d6-403f-bd7d-6ddf1f9d2e3f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:03:01.309Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6e19243f-c0d6-403f-bd7d-6ddf1f9d2e3f ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:03:01.309Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=6e19243f-c0d6-403f-bd7d-6ddf1f9d2e3f ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:03:01.311Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:02.102Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:02.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b164059-42e9-4457-99b4-e52a24c3c9bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:03:02.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b164059-42e9-4457-99b4-e52a24c3c9bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f885edd 2021-05-04T00:03:02.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b164059-42e9-4457-99b4-e52a24c3c9bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:03:02.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b164059-42e9-4457-99b4-e52a24c3c9bb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:03:02.108Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:02.303Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:03:02.603Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:02.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70f2b0ab-90a5-4f2f-b3b9-e538e2513166 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:03:02.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70f2b0ab-90a5-4f2f-b3b9-e538e2513166 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2dd81cc7 2021-05-04T00:03:02.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70f2b0ab-90a5-4f2f-b3b9-e538e2513166 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:03:02.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70f2b0ab-90a5-4f2f-b3b9-e538e2513166 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:03:02.611Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:04.901Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:04.904Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9d61712-40b8-449a-95db-da3aed2cf9da PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:03:04.904Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9d61712-40b8-449a-95db-da3aed2cf9da PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e7a1324 2021-05-04T00:03:04.904Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9d61712-40b8-449a-95db-da3aed2cf9da PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:03:04.904Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9d61712-40b8-449a-95db-da3aed2cf9da PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:03:04.907Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d9d61712-40b8-449a-95db-da3aed2cf9da ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:03:04.907Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d9d61712-40b8-449a-95db-da3aed2cf9da ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:03:04.910Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:03:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:14.500Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d57263f2-7676-46b0-9092-af7b3bd7e180 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:03:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d57263f2-7676-46b0-9092-af7b3bd7e180 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b27439b 2021-05-04T00:03:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d57263f2-7676-46b0-9092-af7b3bd7e180 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:03:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d57263f2-7676-46b0-9092-af7b3bd7e180 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:03:14.509Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:14.602Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:03:16.400Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:16.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afdb86c6-4efb-41eb-a055-914ed8139497 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:03:16.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afdb86c6-4efb-41eb-a055-914ed8139497 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66849c94 2021-05-04T00:03:16.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afdb86c6-4efb-41eb-a055-914ed8139497 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:03:16.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afdb86c6-4efb-41eb-a055-914ed8139497 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:03:16.404Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=afdb86c6-4efb-41eb-a055-914ed8139497 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:03:16.404Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=afdb86c6-4efb-41eb-a055-914ed8139497 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:03:16.406Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:03:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:25.972Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:25.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f67c651-d4bb-42e1-9d78-f10f69b6c743 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:03:25.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f67c651-d4bb-42e1-9d78-f10f69b6c743 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bd78aa7 2021-05-04T00:03:25.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f67c651-d4bb-42e1-9d78-f10f69b6c743 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:03:25.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f67c651-d4bb-42e1-9d78-f10f69b6c743 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:03:25.979Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1f67c651-d4bb-42e1-9d78-f10f69b6c743 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:03:25.979Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1f67c651-d4bb-42e1-9d78-f10f69b6c743 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:03:25.981Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:26.214Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:03:28.509Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T00:03:28.510Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 256000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:30.467Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:30.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1610992-c5da-428f-a4d0-fc7dd9d4f847 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:03:30.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1610992-c5da-428f-a4d0-fc7dd9d4f847 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65da5bca 2021-05-04T00:03:30.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1610992-c5da-428f-a4d0-fc7dd9d4f847 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:03:30.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1610992-c5da-428f-a4d0-fc7dd9d4f847 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:03:30.473Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f1610992-c5da-428f-a4d0-fc7dd9d4f847 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:03:30.473Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f1610992-c5da-428f-a4d0-fc7dd9d4f847 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:03:30.476Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:03:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:03:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:03:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:03:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:03:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:03:37.116Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:03:37.116Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:03:37.118Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:03:37.118Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:03:37.119Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:03:37.120Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:03:37.120Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:03:46.261Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:46.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c911cccc-ae02-45f5-913e-5d79c8bfcf2b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:03:46.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c911cccc-ae02-45f5-913e-5d79c8bfcf2b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6460e771 2021-05-04T00:03:46.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c911cccc-ae02-45f5-913e-5d79c8bfcf2b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:03:46.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c911cccc-ae02-45f5-913e-5d79c8bfcf2b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:03:49.278Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c911cccc-ae02-45f5-913e-5d79c8bfcf2b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T00:03:49.279Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c911cccc-ae02-45f5-913e-5d79c8bfcf2b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T00:03:49.279Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c911cccc-ae02-45f5-913e-5d79c8bfcf2b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:03:49.279Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c911cccc-ae02-45f5-913e-5d79c8bfcf2b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:03:49.283Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:03:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:03:54.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:03:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:03.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:09.428Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.429Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.432Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.432Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7085e21-cd98-47e9-85e1-cabc70d178f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.432Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7085e21-cd98-47e9-85e1-cabc70d178f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@11778388 2021-05-04T00:04:09.432Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7085e21-cd98-47e9-85e1-cabc70d178f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.432Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7085e21-cd98-47e9-85e1-cabc70d178f5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.534Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a7085e21-cd98-47e9-85e1-cabc70d178f5 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:04:09.534Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=a7085e21-cd98-47e9-85e1-cabc70d178f5 ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:04:09.537Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.539Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.539Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa16bf32-6dc1-47c5-9eaf-a273fdc46bc7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.539Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa16bf32-6dc1-47c5-9eaf-a273fdc46bc7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@457cbeb1 2021-05-04T00:04:09.540Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa16bf32-6dc1-47c5-9eaf-a273fdc46bc7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.540Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa16bf32-6dc1-47c5-9eaf-a273fdc46bc7 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.543Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=aa16bf32-6dc1-47c5-9eaf-a273fdc46bc7 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:04:09.543Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=aa16bf32-6dc1-47c5-9eaf-a273fdc46bc7 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:04:09.545Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.547Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3fd6e8-3733-44d3-9800-0995f50881eb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3fd6e8-3733-44d3-9800-0995f50881eb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76b74154 2021-05-04T00:04:09.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3fd6e8-3733-44d3-9800-0995f50881eb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3fd6e8-3733-44d3-9800-0995f50881eb PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.552Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.554Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.554Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b497e61-42d4-492a-b50e-4d7f9b7fb4e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.554Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b497e61-42d4-492a-b50e-4d7f9b7fb4e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5bf1f7c9 2021-05-04T00:04:09.554Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b497e61-42d4-492a-b50e-4d7f9b7fb4e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.554Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b497e61-42d4-492a-b50e-4d7f9b7fb4e0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.560Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.563Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09912aa8-8fa0-4bf4-9207-691198d47272 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09912aa8-8fa0-4bf4-9207-691198d47272 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59c4fd76 2021-05-04T00:04:09.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09912aa8-8fa0-4bf4-9207-691198d47272 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09912aa8-8fa0-4bf4-9207-691198d47272 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.568Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.570Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7a426c3-ea9c-4b42-ab69-d264b935d42d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7a426c3-ea9c-4b42-ab69-d264b935d42d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@796d49c9 2021-05-04T00:04:09.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7a426c3-ea9c-4b42-ab69-d264b935d42d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7a426c3-ea9c-4b42-ab69-d264b935d42d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.575Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.577Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc9d7145-6c62-41f7-866b-01d8c204ec53 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc9d7145-6c62-41f7-866b-01d8c204ec53 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@359c1a51 2021-05-04T00:04:09.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc9d7145-6c62-41f7-866b-01d8c204ec53 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc9d7145-6c62-41f7-866b-01d8c204ec53 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.581Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.583Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.583Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=897bbe60-421a-4f34-8ddc-ba904907b81f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.583Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=897bbe60-421a-4f34-8ddc-ba904907b81f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6187ec94 2021-05-04T00:04:09.583Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=897bbe60-421a-4f34-8ddc-ba904907b81f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.583Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=897bbe60-421a-4f34-8ddc-ba904907b81f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.587Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.589Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.589Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7eb3cfab-4dba-4411-a7b7-080b1514d69d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.589Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7eb3cfab-4dba-4411-a7b7-080b1514d69d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1fd6829f 2021-05-04T00:04:09.589Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7eb3cfab-4dba-4411-a7b7-080b1514d69d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.589Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7eb3cfab-4dba-4411-a7b7-080b1514d69d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.593Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.595Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.595Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c71811dc-7c9e-4d6b-9782-2f53a4638430 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.595Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c71811dc-7c9e-4d6b-9782-2f53a4638430 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ce6a3ad 2021-05-04T00:04:09.595Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c71811dc-7c9e-4d6b-9782-2f53a4638430 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.595Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c71811dc-7c9e-4d6b-9782-2f53a4638430 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.600Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.602Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8adebbaa-fb6e-4e83-a824-fbc2e111097c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8adebbaa-fb6e-4e83-a824-fbc2e111097c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@597d7ef2 2021-05-04T00:04:09.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8adebbaa-fb6e-4e83-a824-fbc2e111097c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8adebbaa-fb6e-4e83-a824-fbc2e111097c PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.607Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.609Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.609Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f44618da-65c1-4912-ae58-96c0b80fecee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.609Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f44618da-65c1-4912-ae58-96c0b80fecee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c0ca05 2021-05-04T00:04:09.609Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f44618da-65c1-4912-ae58-96c0b80fecee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.609Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f44618da-65c1-4912-ae58-96c0b80fecee PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.614Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.615Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34c79679-82a2-441c-a5ac-3199c83e0826 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34c79679-82a2-441c-a5ac-3199c83e0826 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ca96218 2021-05-04T00:04:09.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34c79679-82a2-441c-a5ac-3199c83e0826 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34c79679-82a2-441c-a5ac-3199c83e0826 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.620Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.622Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.622Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=887ae188-4e58-4056-8ff2-5727cfc156d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.622Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=887ae188-4e58-4056-8ff2-5727cfc156d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@156b57eb 2021-05-04T00:04:09.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=887ae188-4e58-4056-8ff2-5727cfc156d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=887ae188-4e58-4056-8ff2-5727cfc156d5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.627Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.628Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.629Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a710d366-145c-4158-a384-7f04b04bd0b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.629Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a710d366-145c-4158-a384-7f04b04bd0b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27d3315a 2021-05-04T00:04:09.629Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a710d366-145c-4158-a384-7f04b04bd0b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.629Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a710d366-145c-4158-a384-7f04b04bd0b8 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.632Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.634Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.634Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb3ff6b1-2bf8-4f17-9fed-5869a70250f3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.634Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb3ff6b1-2bf8-4f17-9fed-5869a70250f3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26c24747 2021-05-04T00:04:09.634Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb3ff6b1-2bf8-4f17-9fed-5869a70250f3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.634Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb3ff6b1-2bf8-4f17-9fed-5869a70250f3 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.638Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.640Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.641Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cdfad9-f853-4b16-9f82-6c0637244d78 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.641Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cdfad9-f853-4b16-9f82-6c0637244d78 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@411c4825 2021-05-04T00:04:09.641Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cdfad9-f853-4b16-9f82-6c0637244d78 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.641Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cdfad9-f853-4b16-9f82-6c0637244d78 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.645Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.646Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e700db2-f481-4812-b4db-3e78d08094b0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e700db2-f481-4812-b4db-3e78d08094b0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e878be2 2021-05-04T00:04:09.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e700db2-f481-4812-b4db-3e78d08094b0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e700db2-f481-4812-b4db-3e78d08094b0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.650Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.652Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c952bed-d33c-4fe2-9fe9-d43806342cfa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c952bed-d33c-4fe2-9fe9-d43806342cfa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30602289 2021-05-04T00:04:09.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c952bed-d33c-4fe2-9fe9-d43806342cfa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c952bed-d33c-4fe2-9fe9-d43806342cfa PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.656Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.658Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e36891c-fc08-492f-ac19-23c6b342f30d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e36891c-fc08-492f-ac19-23c6b342f30d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42ca3bd 2021-05-04T00:04:09.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e36891c-fc08-492f-ac19-23c6b342f30d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e36891c-fc08-492f-ac19-23c6b342f30d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.664Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14c97334-eb24-42b7-b2cf-7257b8bc1a35 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14c97334-eb24-42b7-b2cf-7257b8bc1a35 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27528e7d 2021-05-04T00:04:09.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14c97334-eb24-42b7-b2cf-7257b8bc1a35 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14c97334-eb24-42b7-b2cf-7257b8bc1a35 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.669Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.671Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a89f1b5d-23c4-43d6-ad99-c61480d60327 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a89f1b5d-23c4-43d6-ad99-c61480d60327 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1538d9ee 2021-05-04T00:04:09.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a89f1b5d-23c4-43d6-ad99-c61480d60327 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a89f1b5d-23c4-43d6-ad99-c61480d60327 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.675Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.677Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.677Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd474c1d-b09d-4b2e-9fa5-4eba66fa61b5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.677Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd474c1d-b09d-4b2e-9fa5-4eba66fa61b5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@247dc5aa 2021-05-04T00:04:09.677Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd474c1d-b09d-4b2e-9fa5-4eba66fa61b5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.677Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd474c1d-b09d-4b2e-9fa5-4eba66fa61b5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.681Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.683Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.683Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2f3a9bc-8e21-42ed-83ac-c8650a519c72 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.683Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2f3a9bc-8e21-42ed-83ac-c8650a519c72 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45adbbd3 2021-05-04T00:04:09.683Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2f3a9bc-8e21-42ed-83ac-c8650a519c72 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.683Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2f3a9bc-8e21-42ed-83ac-c8650a519c72 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.687Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.688Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.689Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=310eff49-22e9-4b22-ae53-e1627194650e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.689Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=310eff49-22e9-4b22-ae53-e1627194650e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76bfe47d 2021-05-04T00:04:09.689Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=310eff49-22e9-4b22-ae53-e1627194650e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.689Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=310eff49-22e9-4b22-ae53-e1627194650e PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.692Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.694Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.694Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=526f948f-d54c-4eeb-bcd4-da2d47acd5a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.694Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=526f948f-d54c-4eeb-bcd4-da2d47acd5a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@259ba4ed 2021-05-04T00:04:09.694Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=526f948f-d54c-4eeb-bcd4-da2d47acd5a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.694Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=526f948f-d54c-4eeb-bcd4-da2d47acd5a3 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:09.698Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:09.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffedcf27-c2f9-4553-a074-f50c168382e4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:09.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffedcf27-c2f9-4553-a074-f50c168382e4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f016561 2021-05-04T00:04:09.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffedcf27-c2f9-4553-a074-f50c168382e4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:09.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffedcf27-c2f9-4553-a074-f50c168382e4 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:10.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:10.670Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:10.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d8317f4-8e6b-4678-ae34-620ed52d3454 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:10.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d8317f4-8e6b-4678-ae34-620ed52d3454 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47da2871 2021-05-04T00:04:10.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d8317f4-8e6b-4678-ae34-620ed52d3454 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:10.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d8317f4-8e6b-4678-ae34-620ed52d3454 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:13.699Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:13.701Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:13.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bf196c0-a5ce-4df5-a1b2-280b7221e880 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:13.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bf196c0-a5ce-4df5-a1b2-280b7221e880 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e4fb0fb 2021-05-04T00:04:13.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bf196c0-a5ce-4df5-a1b2-280b7221e880 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:13.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bf196c0-a5ce-4df5-a1b2-280b7221e880 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:15.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:16.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:16.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eda1f121-37ba-4fdd-918c-f3d3c24dbcd1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eda1f121-37ba-4fdd-918c-f3d3c24dbcd1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@360bc913 2021-05-04T00:04:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eda1f121-37ba-4fdd-918c-f3d3c24dbcd1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eda1f121-37ba-4fdd-918c-f3d3c24dbcd1 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:19.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:19.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:19.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=367c8f4e-7536-4b0a-99d2-eca8ef90701f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:19.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=367c8f4e-7536-4b0a-99d2-eca8ef90701f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52c598de 2021-05-04T00:04:19.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=367c8f4e-7536-4b0a-99d2-eca8ef90701f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:19.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=367c8f4e-7536-4b0a-99d2-eca8ef90701f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:22.701Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:22.704Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:22.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fea8c2a-7f9e-45e5-bcf7-988cde0b8ae7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:22.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fea8c2a-7f9e-45e5-bcf7-988cde0b8ae7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44caab57 2021-05-04T00:04:22.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fea8c2a-7f9e-45e5-bcf7-988cde0b8ae7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:22.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fea8c2a-7f9e-45e5-bcf7-988cde0b8ae7 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:25.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:25.667Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:25.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17917c85-4ad2-4aa5-8d73-43ff0ca2309a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:25.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17917c85-4ad2-4aa5-8d73-43ff0ca2309a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7731135d 2021-05-04T00:04:25.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17917c85-4ad2-4aa5-8d73-43ff0ca2309a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:25.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17917c85-4ad2-4aa5-8d73-43ff0ca2309a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:28.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:28.665Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:28.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8e9b857-287e-4907-9179-23960f584020 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:28.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8e9b857-287e-4907-9179-23960f584020 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f9a1ff9 2021-05-04T00:04:28.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8e9b857-287e-4907-9179-23960f584020 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:28.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8e9b857-287e-4907-9179-23960f584020 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:31.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:31.666Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:31.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e462e586-c700-482a-8cc2-633e9d3a010f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:31.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e462e586-c700-482a-8cc2-633e9d3a010f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61e777bc 2021-05-04T00:04:31.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e462e586-c700-482a-8cc2-633e9d3a010f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:31.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e462e586-c700-482a-8cc2-633e9d3a010f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:34.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:34.702Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:34.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=772157cc-d9bd-4dfb-aaec-12fee491fd16 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:34.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=772157cc-d9bd-4dfb-aaec-12fee491fd16 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70e6f8c4 2021-05-04T00:04:34.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=772157cc-d9bd-4dfb-aaec-12fee491fd16 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:34.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=772157cc-d9bd-4dfb-aaec-12fee491fd16 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T00:04:36.203Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:04:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:04:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:04:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:04:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:04:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:04:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:04:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:04:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:04:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:04:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:04:37.117Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:04:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:04:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:04:37.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:04:38.646Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:04:39.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c61c3fd6-25c9-4eb5-8237-9ff67cbaa23d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:39.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c61c3fd6-25c9-4eb5-8237-9ff67cbaa23d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f32b67e 2021-05-04T00:04:39.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c61c3fd6-25c9-4eb5-8237-9ff67cbaa23d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:39.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c61c3fd6-25c9-4eb5-8237-9ff67cbaa23d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:04:39.555Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c61c3fd6-25c9-4eb5-8237-9ff67cbaa23d ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:04:39.555Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=c61c3fd6-25c9-4eb5-8237-9ff67cbaa23d ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:04:42.954Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08d90c51-8a9d-404e-a4a5-51c9fb40d481 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:42.954Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08d90c51-8a9d-404e-a4a5-51c9fb40d481 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@409815b6 2021-05-04T00:04:42.954Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08d90c51-8a9d-404e-a4a5-51c9fb40d481 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:42.954Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08d90c51-8a9d-404e-a4a5-51c9fb40d481 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:04:42.961Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=08d90c51-8a9d-404e-a4a5-51c9fb40d481 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:04:42.961Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=08d90c51-8a9d-404e-a4a5-51c9fb40d481 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:04:44.752Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8b84e25-5a89-4bce-ba83-45b8442b2573 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:44.752Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8b84e25-5a89-4bce-ba83-45b8442b2573 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a98d08e 2021-05-04T00:04:44.752Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8b84e25-5a89-4bce-ba83-45b8442b2573 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:44.753Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8b84e25-5a89-4bce-ba83-45b8442b2573 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:04:48.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89564424-930d-4b40-b609-47da37d80347 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:48.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89564424-930d-4b40-b609-47da37d80347 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d7f2bc1 2021-05-04T00:04:48.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89564424-930d-4b40-b609-47da37d80347 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:48.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89564424-930d-4b40-b609-47da37d80347 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:04:51.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06464dbb-c72b-463f-9706-91e61a6b879a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:51.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06464dbb-c72b-463f-9706-91e61a6b879a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47e4f756 2021-05-04T00:04:51.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06464dbb-c72b-463f-9706-91e61a6b879a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:51.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06464dbb-c72b-463f-9706-91e61a6b879a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:04:51.938Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:51.943Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:51.988Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:51.989Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:51.999Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.001Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.009Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.011Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.014Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.015Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.018Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.019Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.022Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.023Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.026Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.027Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.030Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.030Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.034Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.034Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.039Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.039Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.043Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.044Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.047Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.048Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.051Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.052Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.056Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.056Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.060Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.060Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.064Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.064Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.068Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.068Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.072Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.073Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.076Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.077Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:04:52.082Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:04:52.083Z [UEB-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend connection recovery to UEB Cluster. 2021-05-04T00:04:54.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ce23c23-cab0-4c37-8656-2257fa7bff9a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:54.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ce23c23-cab0-4c37-8656-2257fa7bff9a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47517fb9 2021-05-04T00:04:54.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ce23c23-cab0-4c37-8656-2257fa7bff9a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:54.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ce23c23-cab0-4c37-8656-2257fa7bff9a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:04:54.266Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7ce23c23-cab0-4c37-8656-2257fa7bff9a ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:04:54.266Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7ce23c23-cab0-4c37-8656-2257fa7bff9a ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:04:56.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71700072-9278-4539-a7e7-06a3b58795c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:56.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71700072-9278-4539-a7e7-06a3b58795c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b91cb7d 2021-05-04T00:04:56.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71700072-9278-4539-a7e7-06a3b58795c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:56.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71700072-9278-4539-a7e7-06a3b58795c8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:04:59.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac14058c-0164-459f-849e-354e87d7ff32 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:04:59.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac14058c-0164-459f-849e-354e87d7ff32 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f94daed 2021-05-04T00:04:59.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac14058c-0164-459f-849e-354e87d7ff32 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:04:59.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac14058c-0164-459f-849e-354e87d7ff32 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:00.489Z [qtp580718781-15] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=b4916037-2426-473f-8121-18bb61439615 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:05:00.488499Z InvocationID=253116d2-83e9-4bf8-85b8-aae6197a5422 LogTimestamp=2021-05-04T00:05:00.488651Z Entering 2021-05-04T00:05:00.489Z [qtp580718781-15] INFO o.o.sdc.be.filters.BeServletFilter RequestID=b4916037-2426-473f-8121-18bb61439615 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:05:00.488499Z InvocationID=253116d2-83e9-4bf8-85b8-aae6197a5422 LogTimestamp=2021-05-04T00:05:00.488651Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:05:00.494Z [qtp580718781-15] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=5 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:05:00.489Z InvocationID=d2118246-a5c4-4096-ae2c-1aefe4d9f8f1 LogTimestamp=2021-05-04T00:05:00.494629Z ResponseDesc=OK Exiting. 2021-05-04T00:05:00.495Z [qtp580718781-15] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:05:00.900Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=2009a3a6-903f-451b-b6a6-29aa7d458518 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:05:00.900688Z InvocationID=d132cc0c-7d60-4e9b-aaa7-fea9a99be3b4 LogTimestamp=2021-05-04T00:05:00.900727Z Entering 2021-05-04T00:05:00.901Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=2009a3a6-903f-451b-b6a6-29aa7d458518 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:05:00.900688Z InvocationID=d132cc0c-7d60-4e9b-aaa7-fea9a99be3b4 LogTimestamp=2021-05-04T00:05:00.900727Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:05:00.905Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:05:00.901Z InvocationID=76d5b49d-9206-4063-9ed2-7e29e3e615f1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/5cMwShJqCnAeBc1A will send credentials over a clear channel. 2021-05-04T00:05:00.907Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:05:00.901Z InvocationID=76d5b49d-9206-4063-9ed2-7e29e3e615f1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/5cMwShJqCnAeBc1A (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:05:00.918Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:05:00.901Z InvocationID=76d5b49d-9206-4063-9ed2-7e29e3e615f1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:05:00.923Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "5cMwShJqCnAeBc1A" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:05:00.901Z InvocationID=76d5b49d-9206-4063-9ed2-7e29e3e615f1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/5cMwShJqCnAeBc1A will send credentials over a clear channel. 2021-05-04T00:05:00.924Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "5cMwShJqCnAeBc1A" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:05:00.901Z InvocationID=76d5b49d-9206-4063-9ed2-7e29e3e615f1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/5cMwShJqCnAeBc1A (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:05:00.940Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "5cMwShJqCnAeBc1A" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:05:00.901Z InvocationID=76d5b49d-9206-4063-9ed2-7e29e3e615f1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:05:00.951Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "ves-openapi-manager" API_KEY = "5cMwShJqCnAeBc1A" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=49 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:05:00.901Z InvocationID=76d5b49d-9206-4063-9ed2-7e29e3e615f1 LogTimestamp=2021-05-04T00:05:00.950333Z ResponseDesc=OK Exiting. 2021-05-04T00:05:00.951Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:05:02.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d013f7bc-7747-4612-8d5a-d2558ae1449b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:02.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d013f7bc-7747-4612-8d5a-d2558ae1449b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f2ad40e 2021-05-04T00:05:02.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d013f7bc-7747-4612-8d5a-d2558ae1449b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:02.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d013f7bc-7747-4612-8d5a-d2558ae1449b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:05:05.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:05:05.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9aaf0d2e-4a3d-4c60-b7cb-7a99243ba60d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:05.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9aaf0d2e-4a3d-4c60-b7cb-7a99243ba60d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@134ac4e5 2021-05-04T00:05:05.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9aaf0d2e-4a3d-4c60-b7cb-7a99243ba60d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:05.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9aaf0d2e-4a3d-4c60-b7cb-7a99243ba60d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:08.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1ffd5322-8d99-4be9-a3a6-2a3c85b0088a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:08.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1ffd5322-8d99-4be9-a3a6-2a3c85b0088a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6953fd2 2021-05-04T00:05:08.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1ffd5322-8d99-4be9-a3a6-2a3c85b0088a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:08.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1ffd5322-8d99-4be9-a3a6-2a3c85b0088a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:10.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7479a740-4620-4ffd-8cd3-26238c14c363 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:10.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7479a740-4620-4ffd-8cd3-26238c14c363 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1fcfde5f 2021-05-04T00:05:10.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7479a740-4620-4ffd-8cd3-26238c14c363 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:10.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7479a740-4620-4ffd-8cd3-26238c14c363 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=144c9b68-82e1-4f85-86a9-1ec31b6f0004 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=144c9b68-82e1-4f85-86a9-1ec31b6f0004 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@164ce30c 2021-05-04T00:05:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=144c9b68-82e1-4f85-86a9-1ec31b6f0004 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=144c9b68-82e1-4f85-86a9-1ec31b6f0004 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:17.542Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=706618f0-9f8a-4924-b881-c45d48d57963 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:17.542Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=706618f0-9f8a-4924-b881-c45d48d57963 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53e470cc 2021-05-04T00:05:17.543Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=706618f0-9f8a-4924-b881-c45d48d57963 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:17.543Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=706618f0-9f8a-4924-b881-c45d48d57963 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:19.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab65a5ce-ef11-4fb8-bfd1-5383956c9681 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:19.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab65a5ce-ef11-4fb8-bfd1-5383956c9681 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ca0becd 2021-05-04T00:05:19.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab65a5ce-ef11-4fb8-bfd1-5383956c9681 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:19.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab65a5ce-ef11-4fb8-bfd1-5383956c9681 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:05:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:05:23.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2e7f0dd-9937-4a7b-b02b-3ae7f780095d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:23.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2e7f0dd-9937-4a7b-b02b-3ae7f780095d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d7caa20 2021-05-04T00:05:23.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2e7f0dd-9937-4a7b-b02b-3ae7f780095d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:23.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2e7f0dd-9937-4a7b-b02b-3ae7f780095d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:26.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c148b89-b600-4623-bacc-f46a9f25f8f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:26.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c148b89-b600-4623-bacc-f46a9f25f8f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6cd96e00 2021-05-04T00:05:26.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c148b89-b600-4623-bacc-f46a9f25f8f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:26.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c148b89-b600-4623-bacc-f46a9f25f8f4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:28.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee4490c1-bab2-4b6f-a287-cf39530afbbc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:28.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee4490c1-bab2-4b6f-a287-cf39530afbbc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ccb1147 2021-05-04T00:05:28.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee4490c1-bab2-4b6f-a287-cf39530afbbc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:28.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee4490c1-bab2-4b6f-a287-cf39530afbbc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:31.952Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30d034d5-6732-44ea-82bf-0ebfadee40c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:31.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30d034d5-6732-44ea-82bf-0ebfadee40c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65e492c5 2021-05-04T00:05:31.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30d034d5-6732-44ea-82bf-0ebfadee40c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:31.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30d034d5-6732-44ea-82bf-0ebfadee40c1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:35.173Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:05:35.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:05:35.231Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecdfaeef-8c8f-427b-abf2-96646cff75a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:35.231Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecdfaeef-8c8f-427b-abf2-96646cff75a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1db119f3 2021-05-04T00:05:35.231Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecdfaeef-8c8f-427b-abf2-96646cff75a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:35.232Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecdfaeef-8c8f-427b-abf2-96646cff75a0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:05:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:05:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:05:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:05:37.840Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9edf7b26-7426-4b05-b18c-2459bfa3b583 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:37.840Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9edf7b26-7426-4b05-b18c-2459bfa3b583 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3abfdd2b 2021-05-04T00:05:37.840Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9edf7b26-7426-4b05-b18c-2459bfa3b583 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:37.840Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9edf7b26-7426-4b05-b18c-2459bfa3b583 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:40.907Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5452c258-686c-4661-aa0a-d46ef8d9437f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:40.907Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5452c258-686c-4661-aa0a-d46ef8d9437f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ebaa212 2021-05-04T00:05:40.907Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5452c258-686c-4661-aa0a-d46ef8d9437f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:40.907Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5452c258-686c-4661-aa0a-d46ef8d9437f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:44.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7fdc99d-c7f9-4859-9651-4215cc67ea06 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:44.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7fdc99d-c7f9-4859-9651-4215cc67ea06 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f8975ba 2021-05-04T00:05:44.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7fdc99d-c7f9-4859-9651-4215cc67ea06 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:44.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7fdc99d-c7f9-4859-9651-4215cc67ea06 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:46.785Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=acf11e77-3838-44b1-86a5-47360cd4ad8e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:46.785Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=acf11e77-3838-44b1-86a5-47360cd4ad8e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a629d39 2021-05-04T00:05:46.785Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=acf11e77-3838-44b1-86a5-47360cd4ad8e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:46.785Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=acf11e77-3838-44b1-86a5-47360cd4ad8e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:49.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=844792ea-1034-4d6b-9531-f78a15e2a0a4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:49.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=844792ea-1034-4d6b-9531-f78a15e2a0a4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e704bdb 2021-05-04T00:05:49.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=844792ea-1034-4d6b-9531-f78a15e2a0a4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:49.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=844792ea-1034-4d6b-9531-f78a15e2a0a4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:05:50.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:05:52.772Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:05:53.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0de203a4-b305-4704-aec1-65bd21953d1b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:53.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0de203a4-b305-4704-aec1-65bd21953d1b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45f1de35 2021-05-04T00:05:53.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0de203a4-b305-4704-aec1-65bd21953d1b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:53.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0de203a4-b305-4704-aec1-65bd21953d1b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:56.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9341a16e-2eac-4338-b0b7-df934dfc0c6d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:56.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9341a16e-2eac-4338-b0b7-df934dfc0c6d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31215794 2021-05-04T00:05:56.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9341a16e-2eac-4338-b0b7-df934dfc0c6d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:56.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9341a16e-2eac-4338-b0b7-df934dfc0c6d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:05:58.830Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a43d088-1824-4e69-9dd8-d8dc6a4f96b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:05:58.830Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a43d088-1824-4e69-9dd8-d8dc6a4f96b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f949fdc 2021-05-04T00:05:58.830Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a43d088-1824-4e69-9dd8-d8dc6a4f96b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:05:58.830Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a43d088-1824-4e69-9dd8-d8dc6a4f96b4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:02.425Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46aa88c5-12fd-4492-be71-87289c76662e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:02.425Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46aa88c5-12fd-4492-be71-87289c76662e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63563366 2021-05-04T00:06:02.425Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46aa88c5-12fd-4492-be71-87289c76662e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:02.425Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46aa88c5-12fd-4492-be71-87289c76662e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:06:05.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:06:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94d89029-865e-412d-8bf8-d86ec6d53ebf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94d89029-865e-412d-8bf8-d86ec6d53ebf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@289d494f 2021-05-04T00:06:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94d89029-865e-412d-8bf8-d86ec6d53ebf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94d89029-865e-412d-8bf8-d86ec6d53ebf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:07.854Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1159c08d-ff90-4fdc-8b44-e604a615d142 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:07.854Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1159c08d-ff90-4fdc-8b44-e604a615d142 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13bb363f 2021-05-04T00:06:07.854Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1159c08d-ff90-4fdc-8b44-e604a615d142 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:07.854Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1159c08d-ff90-4fdc-8b44-e604a615d142 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:10.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba149326-6805-4bd0-8f88-df224b84150f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:10.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba149326-6805-4bd0-8f88-df224b84150f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38fa21eb 2021-05-04T00:06:10.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba149326-6805-4bd0-8f88-df224b84150f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:10.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba149326-6805-4bd0-8f88-df224b84150f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:14.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5e8755f-eea1-4273-b7cd-5f8d16997c86 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:14.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5e8755f-eea1-4273-b7cd-5f8d16997c86 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ae5e891 2021-05-04T00:06:14.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5e8755f-eea1-4273-b7cd-5f8d16997c86 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:14.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5e8755f-eea1-4273-b7cd-5f8d16997c86 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02f6b357-f87a-4cad-a060-2199540b1153 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02f6b357-f87a-4cad-a060-2199540b1153 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@287013b3 2021-05-04T00:06:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02f6b357-f87a-4cad-a060-2199540b1153 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02f6b357-f87a-4cad-a060-2199540b1153 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:18.420Z [qtp580718781-18] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=1948eb56-5b15-4905-b7bc-352d6d523030 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:06:18.419776Z InvocationID=f5ef14de-ce16-4515-9601-73b6b4990458 LogTimestamp=2021-05-04T00:06:18.419913Z Entering 2021-05-04T00:06:18.420Z [qtp580718781-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=1948eb56-5b15-4905-b7bc-352d6d523030 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:06:18.419776Z InvocationID=f5ef14de-ce16-4515-9601-73b6b4990458 LogTimestamp=2021-05-04T00:06:18.419913Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:06:18.425Z [qtp580718781-18] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:06:18.421Z InvocationID=c51f80d3-b372-489d-b60d-c208bb540991 LogTimestamp=2021-05-04T00:06:18.425286Z ResponseDesc=OK Exiting. 2021-05-04T00:06:18.425Z [qtp580718781-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:06:18.555Z [qtp580718781-13] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=ab526489-768a-4db6-bfd7-ca1d48c9f0ac InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:06:18.555087Z InvocationID=1d2f3473-f6f8-4aa0-a8ea-1b9fdba9af16 LogTimestamp=2021-05-04T00:06:18.555132Z Entering 2021-05-04T00:06:18.555Z [qtp580718781-13] INFO o.o.sdc.be.filters.BeServletFilter RequestID=ab526489-768a-4db6-bfd7-ca1d48c9f0ac InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:06:18.555087Z InvocationID=1d2f3473-f6f8-4aa0-a8ea-1b9fdba9af16 LogTimestamp=2021-05-04T00:06:18.555132Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:06:18.560Z [qtp580718781-13] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:18.555Z InvocationID=a9b81d85-e768-47ed-9e26-f121ab1e1a5b PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/aW7XAWHx7nUNa9je will send credentials over a clear channel. 2021-05-04T00:06:18.560Z [qtp580718781-13] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:18.555Z InvocationID=a9b81d85-e768-47ed-9e26-f121ab1e1a5b PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/aW7XAWHx7nUNa9je (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:06:18.585Z [qtp580718781-13] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:18.555Z InvocationID=a9b81d85-e768-47ed-9e26-f121ab1e1a5b PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:06:18.594Z [qtp580718781-13] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "aW7XAWHx7nUNa9je" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:18.555Z InvocationID=a9b81d85-e768-47ed-9e26-f121ab1e1a5b PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/aW7XAWHx7nUNa9je will send credentials over a clear channel. 2021-05-04T00:06:18.594Z [qtp580718781-13] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "aW7XAWHx7nUNa9je" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:18.555Z InvocationID=a9b81d85-e768-47ed-9e26-f121ab1e1a5b PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/aW7XAWHx7nUNa9je (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:06:18.605Z [qtp580718781-13] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "aW7XAWHx7nUNa9je" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:18.555Z InvocationID=a9b81d85-e768-47ed-9e26-f121ab1e1a5b PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:06:18.612Z [qtp580718781-13] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "cds" API_KEY = "aW7XAWHx7nUNa9je" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=57 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:18.555Z InvocationID=a9b81d85-e768-47ed-9e26-f121ab1e1a5b LogTimestamp=2021-05-04T00:06:18.612587Z ResponseDesc=OK Exiting. 2021-05-04T00:06:18.613Z [qtp580718781-13] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:06:19.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19718def-f4bc-4732-812d-5340e4e8246d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:19.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19718def-f4bc-4732-812d-5340e4e8246d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62b0c24c 2021-05-04T00:06:19.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19718def-f4bc-4732-812d-5340e4e8246d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:19.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19718def-f4bc-4732-812d-5340e4e8246d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:06:20.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:06:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e553416-5e18-45d9-8470-43f26b540bde PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e553416-5e18-45d9-8470-43f26b540bde PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bac4288 2021-05-04T00:06:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e553416-5e18-45d9-8470-43f26b540bde PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e553416-5e18-45d9-8470-43f26b540bde PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:26.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26389253-fd09-45f2-8168-1afbbd58a800 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:26.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26389253-fd09-45f2-8168-1afbbd58a800 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ec8920c 2021-05-04T00:06:26.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26389253-fd09-45f2-8168-1afbbd58a800 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:26.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26389253-fd09-45f2-8168-1afbbd58a800 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:28.841Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ad981a7-3817-4a6b-b457-1d2d5e27e6d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:28.841Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ad981a7-3817-4a6b-b457-1d2d5e27e6d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@660dfded 2021-05-04T00:06:28.841Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ad981a7-3817-4a6b-b457-1d2d5e27e6d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:28.841Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ad981a7-3817-4a6b-b457-1d2d5e27e6d5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:30.346Z [qtp580718781-11] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=37ba6240-6258-4a90-b174-338100d5f1c9 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:06:30.345675Z InvocationID=c240282c-3ff8-49b9-87e3-b5cc81fa0290 LogTimestamp=2021-05-04T00:06:30.345800Z Entering 2021-05-04T00:06:30.346Z [qtp580718781-11] INFO o.o.sdc.be.filters.BeServletFilter RequestID=37ba6240-6258-4a90-b174-338100d5f1c9 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:06:30.345675Z InvocationID=c240282c-3ff8-49b9-87e3-b5cc81fa0290 LogTimestamp=2021-05-04T00:06:30.345800Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:06:30.349Z [qtp580718781-11] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=2 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:06:30.347Z InvocationID=62a1b377-92f3-427a-857d-fcbdcc03ca28 LogTimestamp=2021-05-04T00:06:30.349285Z ResponseDesc=OK Exiting. 2021-05-04T00:06:30.350Z [qtp580718781-11] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:06:30.434Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=79cb2952-5761-4f7e-b325-b65310f6d5ea InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:06:30.433269Z InvocationID=7061ea96-1768-4a16-b488-cc24cb390195 LogTimestamp=2021-05-04T00:06:30.433319Z Entering 2021-05-04T00:06:30.434Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=79cb2952-5761-4f7e-b325-b65310f6d5ea InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:06:30.433269Z InvocationID=7061ea96-1768-4a16-b488-cc24cb390195 LogTimestamp=2021-05-04T00:06:30.433319Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:06:30.439Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:30.434Z InvocationID=eb0c9547-378e-4df7-996f-2163030b70e1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/At1MUyJ0pG9kH4dh will send credentials over a clear channel. 2021-05-04T00:06:30.439Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:30.434Z InvocationID=eb0c9547-378e-4df7-996f-2163030b70e1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/At1MUyJ0pG9kH4dh (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:06:30.453Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:30.434Z InvocationID=eb0c9547-378e-4df7-996f-2163030b70e1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:06:30.458Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "At1MUyJ0pG9kH4dh" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:30.434Z InvocationID=eb0c9547-378e-4df7-996f-2163030b70e1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/At1MUyJ0pG9kH4dh will send credentials over a clear channel. 2021-05-04T00:06:30.458Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "At1MUyJ0pG9kH4dh" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:30.434Z InvocationID=eb0c9547-378e-4df7-996f-2163030b70e1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/At1MUyJ0pG9kH4dh (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:06:30.469Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "At1MUyJ0pG9kH4dh" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:30.434Z InvocationID=eb0c9547-378e-4df7-996f-2163030b70e1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:06:30.501Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "cds" API_KEY = "At1MUyJ0pG9kH4dh" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=67 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:06:30.434Z InvocationID=eb0c9547-378e-4df7-996f-2163030b70e1 LogTimestamp=2021-05-04T00:06:30.501316Z ResponseDesc=OK Exiting. 2021-05-04T00:06:30.501Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:06:32.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d3179e-0b60-46be-8840-f7cd77cd64e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:32.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d3179e-0b60-46be-8840-f7cd77cd64e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24bf1c2 2021-05-04T00:06:32.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d3179e-0b60-46be-8840-f7cd77cd64e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:32.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d3179e-0b60-46be-8840-f7cd77cd64e1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:06:35.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:06:35.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=764eebf6-f6c4-4aba-966c-82efb65b7669 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:35.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=764eebf6-f6c4-4aba-966c-82efb65b7669 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b1a5b01 2021-05-04T00:06:35.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=764eebf6-f6c4-4aba-966c-82efb65b7669 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:35.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=764eebf6-f6c4-4aba-966c-82efb65b7669 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:06:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:06:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:06:37.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36154100-d6ff-496a-8f13-52c1fc9b148c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:37.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36154100-d6ff-496a-8f13-52c1fc9b148c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6148fc9c 2021-05-04T00:06:37.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36154100-d6ff-496a-8f13-52c1fc9b148c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:37.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36154100-d6ff-496a-8f13-52c1fc9b148c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:40.933Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4659a12-ca30-4908-86b3-05c036a1f937 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:40.933Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4659a12-ca30-4908-86b3-05c036a1f937 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68a387cd 2021-05-04T00:06:40.933Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4659a12-ca30-4908-86b3-05c036a1f937 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:40.933Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4659a12-ca30-4908-86b3-05c036a1f937 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:44.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f93d83a-a547-433a-a642-1d08e0c16c3e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:44.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f93d83a-a547-433a-a642-1d08e0c16c3e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35bbaefa 2021-05-04T00:06:44.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f93d83a-a547-433a-a642-1d08e0c16c3e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:44.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f93d83a-a547-433a-a642-1d08e0c16c3e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:47.253Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb196b8f-ae32-4ae4-bac8-87598b054d7a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:47.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb196b8f-ae32-4ae4-bac8-87598b054d7a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@cc0291f 2021-05-04T00:06:47.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb196b8f-ae32-4ae4-bac8-87598b054d7a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:47.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb196b8f-ae32-4ae4-bac8-87598b054d7a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:49.812Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=153fba49-8ca6-4075-8606-bed3f9aab864 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:49.812Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=153fba49-8ca6-4075-8606-bed3f9aab864 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a4a9807 2021-05-04T00:06:49.812Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=153fba49-8ca6-4075-8606-bed3f9aab864 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:49.812Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=153fba49-8ca6-4075-8606-bed3f9aab864 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:06:50.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:06:52.046Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:06:53.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e52e9016-5a4d-45d6-a622-ee6e6765511f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:53.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e52e9016-5a4d-45d6-a622-ee6e6765511f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ea3ef6c 2021-05-04T00:06:53.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e52e9016-5a4d-45d6-a622-ee6e6765511f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:53.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e52e9016-5a4d-45d6-a622-ee6e6765511f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=753f4b7f-6c69-4eea-b12d-e0a303639ded PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=753f4b7f-6c69-4eea-b12d-e0a303639ded PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3607563c 2021-05-04T00:06:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=753f4b7f-6c69-4eea-b12d-e0a303639ded PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=753f4b7f-6c69-4eea-b12d-e0a303639ded PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:06:58.804Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84029cd1-47e0-4a1c-b3f8-e6ff8acee6f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:06:58.804Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84029cd1-47e0-4a1c-b3f8-e6ff8acee6f8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b820932 2021-05-04T00:06:58.804Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84029cd1-47e0-4a1c-b3f8-e6ff8acee6f8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:06:58.804Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84029cd1-47e0-4a1c-b3f8-e6ff8acee6f8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5242b43-9c1b-424c-8749-c0f6082488e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5242b43-9c1b-424c-8749-c0f6082488e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@457624e5 2021-05-04T00:07:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5242b43-9c1b-424c-8749-c0f6082488e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5242b43-9c1b-424c-8749-c0f6082488e1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:05.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:07:05.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:07:05.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bed6ede-f3e5-4300-b7ff-fd2a866bde93 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:05.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bed6ede-f3e5-4300-b7ff-fd2a866bde93 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ef7ba8 2021-05-04T00:07:05.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bed6ede-f3e5-4300-b7ff-fd2a866bde93 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:05.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bed6ede-f3e5-4300-b7ff-fd2a866bde93 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:07.817Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73e8cce5-2031-45b5-bebe-332afc5b5891 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:07.817Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73e8cce5-2031-45b5-bebe-332afc5b5891 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8557bbf 2021-05-04T00:07:07.817Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73e8cce5-2031-45b5-bebe-332afc5b5891 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:07.817Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73e8cce5-2031-45b5-bebe-332afc5b5891 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:11.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7a689a8-23ac-4134-81f0-906331df5151 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:11.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7a689a8-23ac-4134-81f0-906331df5151 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23b5281e 2021-05-04T00:07:11.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7a689a8-23ac-4134-81f0-906331df5151 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:11.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7a689a8-23ac-4134-81f0-906331df5151 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44d08322-a4f3-429b-9b21-f6fe634a854e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44d08322-a4f3-429b-9b21-f6fe634a854e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57107181 2021-05-04T00:07:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44d08322-a4f3-429b-9b21-f6fe634a854e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44d08322-a4f3-429b-9b21-f6fe634a854e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:17.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ec0b520-c908-4da1-b744-120fd1e77a25 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:17.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ec0b520-c908-4da1-b744-120fd1e77a25 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@285d62f0 2021-05-04T00:07:17.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ec0b520-c908-4da1-b744-120fd1e77a25 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:17.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ec0b520-c908-4da1-b744-120fd1e77a25 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:19.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78469402-7cda-400f-9dc5-d73cedb79994 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:19.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78469402-7cda-400f-9dc5-d73cedb79994 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b767cc7 2021-05-04T00:07:19.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78469402-7cda-400f-9dc5-d73cedb79994 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:19.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78469402-7cda-400f-9dc5-d73cedb79994 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:07:20.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:07:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc6a190d-ae14-419c-a2bc-1d39ccb3aacf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc6a190d-ae14-419c-a2bc-1d39ccb3aacf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@109506f 2021-05-04T00:07:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc6a190d-ae14-419c-a2bc-1d39ccb3aacf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc6a190d-ae14-419c-a2bc-1d39ccb3aacf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:26.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfa882d6-2b88-4b74-afb2-6985071edf8e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:26.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfa882d6-2b88-4b74-afb2-6985071edf8e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55c1a5b6 2021-05-04T00:07:26.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfa882d6-2b88-4b74-afb2-6985071edf8e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:26.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfa882d6-2b88-4b74-afb2-6985071edf8e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:28.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2081ab7-1199-4b9e-afa0-5d2bce7b9eb0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:28.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2081ab7-1199-4b9e-afa0-5d2bce7b9eb0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16c8171c 2021-05-04T00:07:28.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2081ab7-1199-4b9e-afa0-5d2bce7b9eb0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:28.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2081ab7-1199-4b9e-afa0-5d2bce7b9eb0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2c8dbf1-70ae-4249-958e-34ef334a323b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:32.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2c8dbf1-70ae-4249-958e-34ef334a323b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ee0d146 2021-05-04T00:07:32.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2c8dbf1-70ae-4249-958e-34ef334a323b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:32.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2c8dbf1-70ae-4249-958e-34ef334a323b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:07:35.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:07:35.320Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=068416ae-9e2f-4b2d-946c-376bb62a3b25 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:35.320Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=068416ae-9e2f-4b2d-946c-376bb62a3b25 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65a176dd 2021-05-04T00:07:35.320Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=068416ae-9e2f-4b2d-946c-376bb62a3b25 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:35.320Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=068416ae-9e2f-4b2d-946c-376bb62a3b25 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:07:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:07:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:07:37.835Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2d7e232-dbb9-4a09-8aea-2aad8fc98942 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:37.835Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2d7e232-dbb9-4a09-8aea-2aad8fc98942 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d9593ca 2021-05-04T00:07:37.835Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2d7e232-dbb9-4a09-8aea-2aad8fc98942 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:37.835Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2d7e232-dbb9-4a09-8aea-2aad8fc98942 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:41.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=940453ff-d73b-446f-8d27-935386c6df56 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:41.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=940453ff-d73b-446f-8d27-935386c6df56 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b331393 2021-05-04T00:07:41.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=940453ff-d73b-446f-8d27-935386c6df56 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:41.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=940453ff-d73b-446f-8d27-935386c6df56 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:44.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd9f92a3-cead-4f8f-a2c1-99aa027ca3e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:44.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd9f92a3-cead-4f8f-a2c1-99aa027ca3e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43718552 2021-05-04T00:07:44.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd9f92a3-cead-4f8f-a2c1-99aa027ca3e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:44.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd9f92a3-cead-4f8f-a2c1-99aa027ca3e7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T00:07:46.817Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f36e8d54-9ab0-4516-8a88-c68bca9ff0f0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:46.817Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f36e8d54-9ab0-4516-8a88-c68bca9ff0f0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d88147f 2021-05-04T00:07:46.817Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f36e8d54-9ab0-4516-8a88-c68bca9ff0f0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:46.817Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f36e8d54-9ab0-4516-8a88-c68bca9ff0f0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:07:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:07:50.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:07:50.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b821ca2c-6102-4235-9d56-bba6ab4299a4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:50.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b821ca2c-6102-4235-9d56-bba6ab4299a4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c12d9c7 2021-05-04T00:07:50.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b821ca2c-6102-4235-9d56-bba6ab4299a4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:50.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b821ca2c-6102-4235-9d56-bba6ab4299a4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:07:52.073Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:07:53.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d10b6e5-6ef3-473c-85b7-1d7ad053f770 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:53.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d10b6e5-6ef3-473c-85b7-1d7ad053f770 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4fceea5d 2021-05-04T00:07:53.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d10b6e5-6ef3-473c-85b7-1d7ad053f770 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:53.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d10b6e5-6ef3-473c-85b7-1d7ad053f770 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:07:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=993d32a2-6cce-4d89-a47d-aeb762438749 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=993d32a2-6cce-4d89-a47d-aeb762438749 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@dabde3 2021-05-04T00:07:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=993d32a2-6cce-4d89-a47d-aeb762438749 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=993d32a2-6cce-4d89-a47d-aeb762438749 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:07:59.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a79e647-1a34-427c-bf9b-0b844e293ea5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:07:59.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a79e647-1a34-427c-bf9b-0b844e293ea5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d9c6a6b 2021-05-04T00:07:59.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a79e647-1a34-427c-bf9b-0b844e293ea5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:07:59.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a79e647-1a34-427c-bf9b-0b844e293ea5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:02.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3334c237-bd97-4727-b3cc-0da16fe2efd7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:02.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3334c237-bd97-4727-b3cc-0da16fe2efd7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@476a4afe 2021-05-04T00:08:02.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3334c237-bd97-4727-b3cc-0da16fe2efd7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:02.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3334c237-bd97-4727-b3cc-0da16fe2efd7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:05.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:08:05.185Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:08:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04af275b-fd3a-4820-8f8a-37d5bde31d61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04af275b-fd3a-4820-8f8a-37d5bde31d61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f4ecc4c 2021-05-04T00:08:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04af275b-fd3a-4820-8f8a-37d5bde31d61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04af275b-fd3a-4820-8f8a-37d5bde31d61 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:08.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81729225-d782-4c44-9e6a-181f320fb9f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:08.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81729225-d782-4c44-9e6a-181f320fb9f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a2e53e3 2021-05-04T00:08:08.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81729225-d782-4c44-9e6a-181f320fb9f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:08.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81729225-d782-4c44-9e6a-181f320fb9f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:11.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c1d96d7-443e-4666-b902-f16be7982f91 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:11.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c1d96d7-443e-4666-b902-f16be7982f91 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24efabff 2021-05-04T00:08:11.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c1d96d7-443e-4666-b902-f16be7982f91 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:11.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c1d96d7-443e-4666-b902-f16be7982f91 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:14.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=328af9b0-eb58-4ceb-8de3-c7de0ed5c7ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:14.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=328af9b0-eb58-4ceb-8de3-c7de0ed5c7ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31472826 2021-05-04T00:08:14.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=328af9b0-eb58-4ceb-8de3-c7de0ed5c7ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:14.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=328af9b0-eb58-4ceb-8de3-c7de0ed5c7ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f284431f-28a2-4395-97a0-32fae2fc1c27 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f284431f-28a2-4395-97a0-32fae2fc1c27 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3862beee 2021-05-04T00:08:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f284431f-28a2-4395-97a0-32fae2fc1c27 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f284431f-28a2-4395-97a0-32fae2fc1c27 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:08:20.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=184b5dea-aa85-4c26-b830-a527b8b9e14d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:20.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=184b5dea-aa85-4c26-b830-a527b8b9e14d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@608b534a 2021-05-04T00:08:20.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=184b5dea-aa85-4c26-b830-a527b8b9e14d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:20.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=184b5dea-aa85-4c26-b830-a527b8b9e14d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:20.248Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:08:23.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2eeec78-f4cc-446e-bb84-221b07518c1f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:23.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2eeec78-f4cc-446e-bb84-221b07518c1f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3962acf7 2021-05-04T00:08:23.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2eeec78-f4cc-446e-bb84-221b07518c1f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:23.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2eeec78-f4cc-446e-bb84-221b07518c1f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:26.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=645e54f2-5b83-4984-aacb-b37e9ad7e255 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:26.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=645e54f2-5b83-4984-aacb-b37e9ad7e255 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31eca5e5 2021-05-04T00:08:26.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=645e54f2-5b83-4984-aacb-b37e9ad7e255 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:26.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=645e54f2-5b83-4984-aacb-b37e9ad7e255 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:29.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c9d1618-74f8-4cad-9039-f88f86d5095f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:29.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c9d1618-74f8-4cad-9039-f88f86d5095f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1cd4f8f8 2021-05-04T00:08:29.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c9d1618-74f8-4cad-9039-f88f86d5095f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:29.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c9d1618-74f8-4cad-9039-f88f86d5095f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:31.976Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04be585f-e452-4286-a128-bd76e2d66801 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:31.976Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04be585f-e452-4286-a128-bd76e2d66801 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@267ff92b 2021-05-04T00:08:31.976Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04be585f-e452-4286-a128-bd76e2d66801 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:31.976Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04be585f-e452-4286-a128-bd76e2d66801 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:31.979Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=04be585f-e452-4286-a128-bd76e2d66801 PartnerName=SDC.BE I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://sdc-onboarding-be.onap:8445: The target server failed to respond 2021-05-04T00:08:31.979Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=04be585f-e452-4286-a128-bd76e2d66801 PartnerName=SDC.BE Retrying request to {s}->https://sdc-onboarding-be.onap:8445 2021-05-04T00:08:31.987Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=04be585f-e452-4286-a128-bd76e2d66801 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T00:08:31.988Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=04be585f-e452-4286-a128-bd76e2d66801 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T00:08:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:08:35.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccf66c70-f283-469e-8f6c-e08939ad5585 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:35.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccf66c70-f283-469e-8f6c-e08939ad5585 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e41b335 2021-05-04T00:08:35.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccf66c70-f283-469e-8f6c-e08939ad5585 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:35.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccf66c70-f283-469e-8f6c-e08939ad5585 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:35.329Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=ccf66c70-f283-469e-8f6c-e08939ad5585 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T00:08:35.329Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=ccf66c70-f283-469e-8f6c-e08939ad5585 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T00:08:35.636Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:08:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:08:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:08:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:08:38.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c81b466d-b95c-4a09-ace2-160318dbdeea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:38.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c81b466d-b95c-4a09-ace2-160318dbdeea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c06881a 2021-05-04T00:08:38.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c81b466d-b95c-4a09-ace2-160318dbdeea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:38.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c81b466d-b95c-4a09-ace2-160318dbdeea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:41.115Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c81b466d-b95c-4a09-ace2-160318dbdeea ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T00:08:41.116Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c81b466d-b95c-4a09-ace2-160318dbdeea ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T00:08:41.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fdaa9243-e759-4713-8595-6489e8549a6d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:41.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fdaa9243-e759-4713-8595-6489e8549a6d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52862e56 2021-05-04T00:08:41.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fdaa9243-e759-4713-8595-6489e8549a6d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:41.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fdaa9243-e759-4713-8595-6489e8549a6d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:44.733Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=fdaa9243-e759-4713-8595-6489e8549a6d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T00:08:44.733Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=fdaa9243-e759-4713-8595-6489e8549a6d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T00:08:45.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87c894e1-7414-405b-91cc-c0aa6ae57e00 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:45.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87c894e1-7414-405b-91cc-c0aa6ae57e00 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73113e79 2021-05-04T00:08:45.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87c894e1-7414-405b-91cc-c0aa6ae57e00 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:45.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87c894e1-7414-405b-91cc-c0aa6ae57e00 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:48.715Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=87c894e1-7414-405b-91cc-c0aa6ae57e00 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T00:08:48.715Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=87c894e1-7414-405b-91cc-c0aa6ae57e00 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T00:08:49.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6941d6ad-f2f1-4277-9ad3-80b06edb12bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:49.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6941d6ad-f2f1-4277-9ad3-80b06edb12bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e8f1b8c 2021-05-04T00:08:49.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6941d6ad-f2f1-4277-9ad3-80b06edb12bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:49.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6941d6ad-f2f1-4277-9ad3-80b06edb12bd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:08:52.140Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:08:52.306Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6941d6ad-f2f1-4277-9ad3-80b06edb12bd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T00:08:52.307Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6941d6ad-f2f1-4277-9ad3-80b06edb12bd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T00:08:52.699Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c630f7d4-ddfd-485f-945a-ac8b5750c0e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:52.699Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c630f7d4-ddfd-485f-945a-ac8b5750c0e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63095a8 2021-05-04T00:08:52.699Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c630f7d4-ddfd-485f-945a-ac8b5750c0e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:52.699Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c630f7d4-ddfd-485f-945a-ac8b5750c0e6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:55.711Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c630f7d4-ddfd-485f-945a-ac8b5750c0e6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T00:08:55.711Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c630f7d4-ddfd-485f-945a-ac8b5750c0e6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T00:08:56.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fc51060-e00a-45f8-aadb-9d2493bdbeb1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:56.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fc51060-e00a-45f8-aadb-9d2493bdbeb1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e3463b8 2021-05-04T00:08:56.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fc51060-e00a-45f8-aadb-9d2493bdbeb1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:56.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fc51060-e00a-45f8-aadb-9d2493bdbeb1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:08:58.204Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:08:59.327Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8fc51060-e00a-45f8-aadb-9d2493bdbeb1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T00:08:59.327Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8fc51060-e00a-45f8-aadb-9d2493bdbeb1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T00:08:59.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=786dd38e-7193-49c2-8973-5266b0c0d184 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:08:59.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=786dd38e-7193-49c2-8973-5266b0c0d184 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64ceeff5 2021-05-04T00:08:59.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=786dd38e-7193-49c2-8973-5266b0c0d184 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:08:59.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=786dd38e-7193-49c2-8973-5266b0c0d184 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:02.196Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=786dd38e-7193-49c2-8973-5266b0c0d184 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:09:02.196Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=786dd38e-7193-49c2-8973-5266b0c0d184 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:09:03.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db829e44-91c3-4847-843f-aea00437c269 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:03.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db829e44-91c3-4847-843f-aea00437c269 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b2b5433 2021-05-04T00:09:03.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db829e44-91c3-4847-843f-aea00437c269 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:03.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db829e44-91c3-4847-843f-aea00437c269 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:04.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=85eb0e96-990b-4ed2-a0a4-93d877348b3e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:04.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=85eb0e96-990b-4ed2-a0a4-93d877348b3e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20cb6dbf 2021-05-04T00:09:04.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=85eb0e96-990b-4ed2-a0a4-93d877348b3e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:04.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=85eb0e96-990b-4ed2-a0a4-93d877348b3e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:09:05.221Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:09:05.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cf635c6-9648-49d2-8066-51f611e4c4b7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:05.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cf635c6-9648-49d2-8066-51f611e4c4b7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33e98f14 2021-05-04T00:09:05.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cf635c6-9648-49d2-8066-51f611e4c4b7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:05.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cf635c6-9648-49d2-8066-51f611e4c4b7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:08.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b5ffaff-f11a-40c0-ab69-f8ab71efec56 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:08.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b5ffaff-f11a-40c0-ab69-f8ab71efec56 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b99417e 2021-05-04T00:09:08.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b5ffaff-f11a-40c0-ab69-f8ab71efec56 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:08.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b5ffaff-f11a-40c0-ab69-f8ab71efec56 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4990f2dd-eb84-48a3-8a7e-cb421d274165 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4990f2dd-eb84-48a3-8a7e-cb421d274165 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a8208f2 2021-05-04T00:09:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4990f2dd-eb84-48a3-8a7e-cb421d274165 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4990f2dd-eb84-48a3-8a7e-cb421d274165 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:14.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da332aef-1aa9-40ef-8b1a-b1cd9b2c85c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:14.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da332aef-1aa9-40ef-8b1a-b1cd9b2c85c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@11df2688 2021-05-04T00:09:14.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da332aef-1aa9-40ef-8b1a-b1cd9b2c85c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:14.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da332aef-1aa9-40ef-8b1a-b1cd9b2c85c1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:17.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=445786e7-0e87-423f-b33d-01569ae6c6fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=445786e7-0e87-423f-b33d-01569ae6c6fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e319a88 2021-05-04T00:09:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=445786e7-0e87-423f-b33d-01569ae6c6fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=445786e7-0e87-423f-b33d-01569ae6c6fe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:09:20.182Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:09:20.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b57864b-d4d1-4f04-8d86-df3261a85177 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:20.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b57864b-d4d1-4f04-8d86-df3261a85177 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f12b481 2021-05-04T00:09:20.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b57864b-d4d1-4f04-8d86-df3261a85177 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:20.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b57864b-d4d1-4f04-8d86-df3261a85177 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:23.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9212ae0-fa23-4e70-8317-acb256245179 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:23.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9212ae0-fa23-4e70-8317-acb256245179 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@252df209 2021-05-04T00:09:23.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9212ae0-fa23-4e70-8317-acb256245179 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:23.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9212ae0-fa23-4e70-8317-acb256245179 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:26.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b66568e2-6967-4f39-9452-98b667aeb4d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:26.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b66568e2-6967-4f39-9452-98b667aeb4d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d2e7b42 2021-05-04T00:09:26.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b66568e2-6967-4f39-9452-98b667aeb4d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:26.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b66568e2-6967-4f39-9452-98b667aeb4d6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:29.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adaefb16-be48-41b2-9e01-12dc40e79eeb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:29.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adaefb16-be48-41b2-9e01-12dc40e79eeb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f295db5 2021-05-04T00:09:29.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adaefb16-be48-41b2-9e01-12dc40e79eeb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:29.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adaefb16-be48-41b2-9e01-12dc40e79eeb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9755a0d1-3e9c-4939-b455-a0d3ec0b730b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9755a0d1-3e9c-4939-b455-a0d3ec0b730b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65d5526d 2021-05-04T00:09:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9755a0d1-3e9c-4939-b455-a0d3ec0b730b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9755a0d1-3e9c-4939-b455-a0d3ec0b730b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:35.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:09:35.184Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:09:35.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=981a20f1-769c-48ad-aae5-e0279ed022ae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:35.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=981a20f1-769c-48ad-aae5-e0279ed022ae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f225964 2021-05-04T00:09:35.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=981a20f1-769c-48ad-aae5-e0279ed022ae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=981a20f1-769c-48ad-aae5-e0279ed022ae PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:09:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:09:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:09:38.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb1a5276-45e2-4db0-a123-db933332c363 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:38.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb1a5276-45e2-4db0-a123-db933332c363 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1168c3a9 2021-05-04T00:09:38.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb1a5276-45e2-4db0-a123-db933332c363 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:38.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb1a5276-45e2-4db0-a123-db933332c363 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:41.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b82018d-3bbb-4bce-9654-3331b3e1a4aa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:41.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b82018d-3bbb-4bce-9654-3331b3e1a4aa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74c9e0c4 2021-05-04T00:09:41.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b82018d-3bbb-4bce-9654-3331b3e1a4aa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:41.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b82018d-3bbb-4bce-9654-3331b3e1a4aa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:44.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fec260f-95e8-4a80-88b8-4495ea7e7d91 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:44.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fec260f-95e8-4a80-88b8-4495ea7e7d91 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b40820a 2021-05-04T00:09:44.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fec260f-95e8-4a80-88b8-4495ea7e7d91 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:44.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fec260f-95e8-4a80-88b8-4495ea7e7d91 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:47.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f55d45d0-16f0-446f-8773-6ca7119b0c65 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:47.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f55d45d0-16f0-446f-8773-6ca7119b0c65 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@694157f 2021-05-04T00:09:47.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f55d45d0-16f0-446f-8773-6ca7119b0c65 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:47.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f55d45d0-16f0-446f-8773-6ca7119b0c65 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:09:50.192Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:09:50.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=661f331f-6d23-4604-b09e-74876feff8f2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:50.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=661f331f-6d23-4604-b09e-74876feff8f2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5136eb8f 2021-05-04T00:09:50.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=661f331f-6d23-4604-b09e-74876feff8f2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:50.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=661f331f-6d23-4604-b09e-74876feff8f2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:52.080Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:09:53.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94388fd0-a888-465b-a64b-7685dcd88f46 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:53.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94388fd0-a888-465b-a64b-7685dcd88f46 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a42fead 2021-05-04T00:09:53.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94388fd0-a888-465b-a64b-7685dcd88f46 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:53.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94388fd0-a888-465b-a64b-7685dcd88f46 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1047f1b7-76f7-4835-a9b4-9be7b3bd363a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1047f1b7-76f7-4835-a9b4-9be7b3bd363a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@552d713d 2021-05-04T00:09:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1047f1b7-76f7-4835-a9b4-9be7b3bd363a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1047f1b7-76f7-4835-a9b4-9be7b3bd363a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:09:59.226Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36e893e3-95a4-482e-a666-b3139c33bf3f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:09:59.226Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36e893e3-95a4-482e-a666-b3139c33bf3f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73b4a90a 2021-05-04T00:09:59.226Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36e893e3-95a4-482e-a666-b3139c33bf3f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:09:59.226Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36e893e3-95a4-482e-a666-b3139c33bf3f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:02.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1088329-79a6-4ea9-b085-34d0e18956c3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:02.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1088329-79a6-4ea9-b085-34d0e18956c3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15b84e60 2021-05-04T00:10:02.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1088329-79a6-4ea9-b085-34d0e18956c3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:02.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1088329-79a6-4ea9-b085-34d0e18956c3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:10:05.220Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:10:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=930ddd2e-cfbe-42db-8b0b-a31eeeada991 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=930ddd2e-cfbe-42db-8b0b-a31eeeada991 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68727dec 2021-05-04T00:10:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=930ddd2e-cfbe-42db-8b0b-a31eeeada991 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=930ddd2e-cfbe-42db-8b0b-a31eeeada991 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:08.058Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node018cdewz78nw0817ms88g0uctc7197 2021-05-04T00:10:08.058Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0jl4texlonmpab47xh0g6t48m191 2021-05-04T00:10:08.058Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0pz6uvh1nzjbq1lnfeul7ftgbb169 2021-05-04T00:10:08.058Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01m0oiaoqy8cn71x99td76wtgwi207 2021-05-04T00:10:08.058Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01dysajxithcvwcbsyevm195x9170 2021-05-04T00:10:08.058Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node017y9ziuqiwzq5q5zh9zedk91c206 2021-05-04T00:10:08.058Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0qb95u603puxbty9ln1eula9a182 2021-05-04T00:10:08.058Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node09tye7nkhr23l10tnetx96ci59172 2021-05-04T00:10:08.058Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node016iaz9x3wrmb01ok4tu495acb6180 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ncasf9gwl8wr64iw49haebsj216 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0krvb1gyiqszpv30472o0afar221 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0cabqbi6tl8nu1uo5twif85htg189 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0t98gsx200627dmpdr4xkzg2n205 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0clhj74cemi9huipl0jlmps9f209 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0lll6oj4ql0v01hxfrrv9k9s6176 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01g68869drlddb1jbm0y9neztg6177 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01wrj3xur3z9v7pbgmytghynza194 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0frrz2jjezsecr3ek7u39yhne201 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node08ss91vm497k4tapirtbspj70215 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0j7yuxaednkya1flpl04fmok5n173 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01hr6dipeyyu2n2b8yxx4xw77b174 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0rednxgmvice4ayf9305ipoca203 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01cicfbt36yhf35pmjl5ho4y1n181 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0inwwj8pssb4g1a9qn0qnpopva186 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01czmh8mfd35531b0p7hubivoqo208 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ou63i1nn6vx71gvyf3hgd3057204 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0tpqck64s8uxr8o5amhwbj6pk183 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ztbkr8jcl76llj64z21ixx26175 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01hzc9vbseqral1l1pkbjtdu1f5195 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01bobaw47kdw6im52t3ffprqsd188 2021-05-04T00:10:08.059Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01gn8u4bfp0wf81bt837wbr4d9o200 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01czdeusd3xnew14xufq4d1wp7l192 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node093zr6n6d0b11bfevjsqij5qy211 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0l4mbgejf95961dpxbf1v1ge0e218 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0zv1xxv7vibq7lm0vaxclbwhs187 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0fwg4y7v00eupxto57m87vvul178 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node04cf3gv07jfy1mtmy4vmkj2l7217 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0a1yg5fe395drao1cdurseau2198 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0npw3a6sfben3dj8l1cglrigw171 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01jc72f36jxps1lo29ufllpram210 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01nvvvz7ncrwxip24arpojty6s212 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node09qs5ddupq837a6f1gp4glf0o199 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0gpz823eqdshf10z1tnrfi9opn179 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node052240dt6rz8yx5hb1xbr3mmq202 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01otavb2kk9sdb12z3qwbqti0wv214 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0144mgry5mrh71vk8k9a27df81185 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node058n701h6bjpp1d4n85qcjwbyu193 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01dwvho7txy6d14jtmjarwzj5t190 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0a7ne4jtq252y4y5qoam7pkei184 2021-05-04T00:10:08.060Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0fnpn2xxngjsn13l19gpk6uqab196 2021-05-04T00:10:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c176f59-5889-43f9-988a-484c174b75fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c176f59-5889-43f9-988a-484c174b75fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56e85d46 2021-05-04T00:10:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c176f59-5889-43f9-988a-484c174b75fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c176f59-5889-43f9-988a-484c174b75fa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:11.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46f2776a-5126-4877-97d1-a33dd2842e67 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:11.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46f2776a-5126-4877-97d1-a33dd2842e67 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@424268 2021-05-04T00:10:11.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46f2776a-5126-4877-97d1-a33dd2842e67 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:11.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46f2776a-5126-4877-97d1-a33dd2842e67 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ed31f26-9320-4a59-8223-6937e9f16119 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ed31f26-9320-4a59-8223-6937e9f16119 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57ded74e 2021-05-04T00:10:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ed31f26-9320-4a59-8223-6937e9f16119 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ed31f26-9320-4a59-8223-6937e9f16119 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:17.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a01af9bb-e6df-4298-9238-1b0c5a5915dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:17.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a01af9bb-e6df-4298-9238-1b0c5a5915dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c384d44 2021-05-04T00:10:17.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a01af9bb-e6df-4298-9238-1b0c5a5915dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:17.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a01af9bb-e6df-4298-9238-1b0c5a5915dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:10:20.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f05e1b40-f38e-4a1c-935b-86004c7df32d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:20.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f05e1b40-f38e-4a1c-935b-86004c7df32d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38ae37e8 2021-05-04T00:10:20.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f05e1b40-f38e-4a1c-935b-86004c7df32d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:20.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f05e1b40-f38e-4a1c-935b-86004c7df32d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:20.526Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:10:23.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1332919d-4395-4787-9f6b-ac0b89e091a9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:23.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1332919d-4395-4787-9f6b-ac0b89e091a9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a6d6397 2021-05-04T00:10:23.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1332919d-4395-4787-9f6b-ac0b89e091a9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:23.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1332919d-4395-4787-9f6b-ac0b89e091a9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:26.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39eb42c0-d809-464c-9a7b-6137f974c168 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:26.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39eb42c0-d809-464c-9a7b-6137f974c168 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b28b533 2021-05-04T00:10:26.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39eb42c0-d809-464c-9a7b-6137f974c168 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:26.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39eb42c0-d809-464c-9a7b-6137f974c168 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:29.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29b6c607-86a1-4b46-86f8-41c9459021d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:29.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29b6c607-86a1-4b46-86f8-41c9459021d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65149b7f 2021-05-04T00:10:29.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29b6c607-86a1-4b46-86f8-41c9459021d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:29.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29b6c607-86a1-4b46-86f8-41c9459021d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:32.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c735b5a-06ec-4902-a307-7132d61baecf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:32.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c735b5a-06ec-4902-a307-7132d61baecf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c4ce142 2021-05-04T00:10:32.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c735b5a-06ec-4902-a307-7132d61baecf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:32.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c735b5a-06ec-4902-a307-7132d61baecf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:10:35.387Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:10:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45ccf5a9-04e1-495c-b769-d850af357e6f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45ccf5a9-04e1-495c-b769-d850af357e6f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6dfc5e87 2021-05-04T00:10:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45ccf5a9-04e1-495c-b769-d850af357e6f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45ccf5a9-04e1-495c-b769-d850af357e6f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:10:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:10:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:10:38.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ab2b44-ab5d-4d79-9b39-13e70c1934b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:38.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ab2b44-ab5d-4d79-9b39-13e70c1934b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23b3ff26 2021-05-04T00:10:38.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ab2b44-ab5d-4d79-9b39-13e70c1934b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:38.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ab2b44-ab5d-4d79-9b39-13e70c1934b9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:41.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce3b4c98-8393-440d-b7db-2fc8aab88535 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:41.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce3b4c98-8393-440d-b7db-2fc8aab88535 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@623d474c 2021-05-04T00:10:41.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce3b4c98-8393-440d-b7db-2fc8aab88535 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:41.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce3b4c98-8393-440d-b7db-2fc8aab88535 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:44.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27986b1a-068d-4e49-8416-df79358f3de7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:44.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27986b1a-068d-4e49-8416-df79358f3de7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29b0ecb8 2021-05-04T00:10:44.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27986b1a-068d-4e49-8416-df79358f3de7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:44.548Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27986b1a-068d-4e49-8416-df79358f3de7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:47.093Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=287589da-469d-477f-bf38-01c9ea769043 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:47.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=287589da-469d-477f-bf38-01c9ea769043 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2637876f 2021-05-04T00:10:47.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=287589da-469d-477f-bf38-01c9ea769043 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:47.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=287589da-469d-477f-bf38-01c9ea769043 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:50.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:10:50.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:10:50.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9a00802-ada2-4b4f-976e-4ff62f992507 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:50.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9a00802-ada2-4b4f-976e-4ff62f992507 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a411640 2021-05-04T00:10:50.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9a00802-ada2-4b4f-976e-4ff62f992507 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:50.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9a00802-ada2-4b4f-976e-4ff62f992507 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:52.113Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:10:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fdf6bab-c7ed-4f16-8eed-e56d3c733a4b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fdf6bab-c7ed-4f16-8eed-e56d3c733a4b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@124e2563 2021-05-04T00:10:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fdf6bab-c7ed-4f16-8eed-e56d3c733a4b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fdf6bab-c7ed-4f16-8eed-e56d3c733a4b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd7bdbb3-5031-45cd-8193-7c471a0885cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd7bdbb3-5031-45cd-8193-7c471a0885cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c35cebe 2021-05-04T00:10:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd7bdbb3-5031-45cd-8193-7c471a0885cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd7bdbb3-5031-45cd-8193-7c471a0885cb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:59.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ce5f2c2-1afd-4c10-8b41-af7c223b6e6d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:10:59.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ce5f2c2-1afd-4c10-8b41-af7c223b6e6d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26233aa8 2021-05-04T00:10:59.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ce5f2c2-1afd-4c10-8b41-af7c223b6e6d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:10:59.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ce5f2c2-1afd-4c10-8b41-af7c223b6e6d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:10:59.525Z [qtp580718781-20] WARN o.o.s.e.AbstractSdncException RequestID=null ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName=/healthCheck TargetServiceName=null EntryTimestamp=2021-05-04T00:10:21.791Z InvocationID=1fe4475b-67bc-47d2-86c6-3ef82c577eaa ErrorCode=900 PartnerName=Apache-HttpClient/4.5.3 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Received less parameters than expected for error with messageId SVC4003, expected: 1, actual: 0. Missing parameters are padded with null values. 2021-05-04T00:10:59.608Z [qtp580718781-20] INFO o.o.s.b.s.e.ComponentExceptionMapper AuditMessage=ACTION = "Access" USER = "(test)" STATUS = "404" DESC = "SVC4003: Error: User 'null' was not found." RequestID=null PartnerName=Apache-HttpClient/4.5.3 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName=/healthCheck TargetServiceName=null EntryTimestamp=2021-05-04T00:10:21.791Z InvocationID=1fe4475b-67bc-47d2-86c6-3ef82c577eaa ErrorCode=900 Error during request filter= USER_NOT_FOUND 2021-05-04T00:11:02.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d29da63-11f5-43a2-b6e8-a2df88540e93 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:02.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d29da63-11f5-43a2-b6e8-a2df88540e93 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3442db3a 2021-05-04T00:11:02.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d29da63-11f5-43a2-b6e8-a2df88540e93 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:02.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d29da63-11f5-43a2-b6e8-a2df88540e93 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:11:05.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:11:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e9e7c6c-a351-4123-ae40-ad550f55870a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e9e7c6c-a351-4123-ae40-ad550f55870a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57c495ec 2021-05-04T00:11:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e9e7c6c-a351-4123-ae40-ad550f55870a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e9e7c6c-a351-4123-ae40-ad550f55870a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8223719-d3cc-4fb5-bd94-b69036a82151 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8223719-d3cc-4fb5-bd94-b69036a82151 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4516e194 2021-05-04T00:11:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8223719-d3cc-4fb5-bd94-b69036a82151 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8223719-d3cc-4fb5-bd94-b69036a82151 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:11.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b290693-def7-4a26-8b87-d8c249757d86 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:11.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b290693-def7-4a26-8b87-d8c249757d86 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7adf75e4 2021-05-04T00:11:11.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b290693-def7-4a26-8b87-d8c249757d86 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:11.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b290693-def7-4a26-8b87-d8c249757d86 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:14.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3514199-dcd9-4d26-bb9a-535307e10183 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:14.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3514199-dcd9-4d26-bb9a-535307e10183 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58e6fa8e 2021-05-04T00:11:14.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3514199-dcd9-4d26-bb9a-535307e10183 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:14.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3514199-dcd9-4d26-bb9a-535307e10183 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:17.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28bc7c8c-9163-4a37-821b-8d98b2924241 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:17.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28bc7c8c-9163-4a37-821b-8d98b2924241 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2bfe6f8b 2021-05-04T00:11:17.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28bc7c8c-9163-4a37-821b-8d98b2924241 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:17.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28bc7c8c-9163-4a37-821b-8d98b2924241 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:20.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=962905b9-0b95-4255-a809-c781d6b24324 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:20.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=962905b9-0b95-4255-a809-c781d6b24324 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59963121 2021-05-04T00:11:20.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=962905b9-0b95-4255-a809-c781d6b24324 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:20.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=962905b9-0b95-4255-a809-c781d6b24324 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:11:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:11:23.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93d8849a-892d-4279-9d40-1db51504a5f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:23.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93d8849a-892d-4279-9d40-1db51504a5f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58e71fa2 2021-05-04T00:11:23.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93d8849a-892d-4279-9d40-1db51504a5f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:23.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93d8849a-892d-4279-9d40-1db51504a5f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:26.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22f18d34-d8d1-4612-b2a0-38f4c34a049b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:26.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22f18d34-d8d1-4612-b2a0-38f4c34a049b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37baacba 2021-05-04T00:11:26.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22f18d34-d8d1-4612-b2a0-38f4c34a049b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:26.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22f18d34-d8d1-4612-b2a0-38f4c34a049b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:29.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7e4e331-e4a1-4241-a9e4-1b15a61295df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:29.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7e4e331-e4a1-4241-a9e4-1b15a61295df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6fe01ff1 2021-05-04T00:11:29.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7e4e331-e4a1-4241-a9e4-1b15a61295df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:29.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7e4e331-e4a1-4241-a9e4-1b15a61295df PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:32.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfecbeb-8e6f-48b9-ae33-3b9a69eb3604 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:32.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfecbeb-8e6f-48b9-ae33-3b9a69eb3604 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7172415 2021-05-04T00:11:32.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfecbeb-8e6f-48b9-ae33-3b9a69eb3604 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:32.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfecbeb-8e6f-48b9-ae33-3b9a69eb3604 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:11:35.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:11:35.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4851f87-7879-4d9f-a657-6869318fa2f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:35.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4851f87-7879-4d9f-a657-6869318fa2f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@676d315a 2021-05-04T00:11:35.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4851f87-7879-4d9f-a657-6869318fa2f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:35.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4851f87-7879-4d9f-a657-6869318fa2f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:37.109Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:11:37.110Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:11:37.110Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:11:38.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b3b73ad-c869-4073-a0a0-f377859a93a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:38.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b3b73ad-c869-4073-a0a0-f377859a93a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5509e71 2021-05-04T00:11:38.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b3b73ad-c869-4073-a0a0-f377859a93a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:38.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b3b73ad-c869-4073-a0a0-f377859a93a0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:41.230Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd362976-7bdd-4c0f-92d8-5471a02f4d12 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:41.230Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd362976-7bdd-4c0f-92d8-5471a02f4d12 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6aa006bd 2021-05-04T00:11:41.230Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd362976-7bdd-4c0f-92d8-5471a02f4d12 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:41.230Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd362976-7bdd-4c0f-92d8-5471a02f4d12 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:44.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a256ad-c0d5-441b-9233-0279eeb307de PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:44.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a256ad-c0d5-441b-9233-0279eeb307de PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5429b070 2021-05-04T00:11:44.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a256ad-c0d5-441b-9233-0279eeb307de PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:44.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a256ad-c0d5-441b-9233-0279eeb307de PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:46.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4996bbbc-1f1b-4dd8-bda5-207d0e7b06f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:46.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4996bbbc-1f1b-4dd8-bda5-207d0e7b06f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ac0ac4e 2021-05-04T00:11:46.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4996bbbc-1f1b-4dd8-bda5-207d0e7b06f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:46.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4996bbbc-1f1b-4dd8-bda5-207d0e7b06f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:11:50.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:11:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9ccef16-6e96-45d2-bffb-329f6b8e33e2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9ccef16-6e96-45d2-bffb-329f6b8e33e2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b620271 2021-05-04T00:11:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9ccef16-6e96-45d2-bffb-329f6b8e33e2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9ccef16-6e96-45d2-bffb-329f6b8e33e2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:52.051Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:11:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79a424c2-1be4-4fe1-b755-31ca6dd2c166 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79a424c2-1be4-4fe1-b755-31ca6dd2c166 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@319d624e 2021-05-04T00:11:53.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79a424c2-1be4-4fe1-b755-31ca6dd2c166 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:53.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79a424c2-1be4-4fe1-b755-31ca6dd2c166 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=665a30d8-4c96-4573-b1b9-830f081fea1a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=665a30d8-4c96-4573-b1b9-830f081fea1a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28aeee3d 2021-05-04T00:11:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=665a30d8-4c96-4573-b1b9-830f081fea1a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=665a30d8-4c96-4573-b1b9-830f081fea1a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:11:59.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0473554-125e-46b3-801e-3514c628f3f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:11:59.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0473554-125e-46b3-801e-3514c628f3f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77d395ba 2021-05-04T00:11:59.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0473554-125e-46b3-801e-3514c628f3f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:11:59.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0473554-125e-46b3-801e-3514c628f3f6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:02.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1809dc9e-927d-4efb-82a6-adc23988d93d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:02.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1809dc9e-927d-4efb-82a6-adc23988d93d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c6970bb 2021-05-04T00:12:02.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1809dc9e-927d-4efb-82a6-adc23988d93d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:02.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1809dc9e-927d-4efb-82a6-adc23988d93d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:12:05.184Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:12:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e9b6859-1a74-4143-b0ee-57fdad532970 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e9b6859-1a74-4143-b0ee-57fdad532970 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66bcea6a 2021-05-04T00:12:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e9b6859-1a74-4143-b0ee-57fdad532970 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e9b6859-1a74-4143-b0ee-57fdad532970 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:08.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30d9949e-f021-49af-b609-c95a5f28ce7d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:08.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30d9949e-f021-49af-b609-c95a5f28ce7d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@393c3f86 2021-05-04T00:12:08.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30d9949e-f021-49af-b609-c95a5f28ce7d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:08.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30d9949e-f021-49af-b609-c95a5f28ce7d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:11.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54ef3408-260a-459d-b2e8-1f4139146439 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:11.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54ef3408-260a-459d-b2e8-1f4139146439 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5229035e 2021-05-04T00:12:11.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54ef3408-260a-459d-b2e8-1f4139146439 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:11.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54ef3408-260a-459d-b2e8-1f4139146439 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04d05496-27b6-4e3c-bfa4-e0b52fe02b96 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04d05496-27b6-4e3c-bfa4-e0b52fe02b96 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25b24d08 2021-05-04T00:12:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04d05496-27b6-4e3c-bfa4-e0b52fe02b96 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04d05496-27b6-4e3c-bfa4-e0b52fe02b96 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:17.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26104874-6f60-4110-b666-f495739a5d4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:17.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26104874-6f60-4110-b666-f495739a5d4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24dc1c6e 2021-05-04T00:12:17.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26104874-6f60-4110-b666-f495739a5d4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:17.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26104874-6f60-4110-b666-f495739a5d4a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:12:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:12:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adb63656-0b1b-4e05-8f63-9f134841e9fc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adb63656-0b1b-4e05-8f63-9f134841e9fc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cc4c77c 2021-05-04T00:12:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adb63656-0b1b-4e05-8f63-9f134841e9fc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adb63656-0b1b-4e05-8f63-9f134841e9fc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=63da861d-af87-4aac-8ab1-72fde670e63e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=63da861d-af87-4aac-8ab1-72fde670e63e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66ed0849 2021-05-04T00:12:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=63da861d-af87-4aac-8ab1-72fde670e63e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=63da861d-af87-4aac-8ab1-72fde670e63e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:26.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62df5dcc-4c29-4536-bf8a-b7cf545d0224 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:26.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62df5dcc-4c29-4536-bf8a-b7cf545d0224 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5180d391 2021-05-04T00:12:26.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62df5dcc-4c29-4536-bf8a-b7cf545d0224 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:26.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62df5dcc-4c29-4536-bf8a-b7cf545d0224 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:30.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02b0f904-d81c-4e9b-88c8-2af7a5871a62 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:30.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02b0f904-d81c-4e9b-88c8-2af7a5871a62 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15fb9aa3 2021-05-04T00:12:30.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02b0f904-d81c-4e9b-88c8-2af7a5871a62 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:30.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02b0f904-d81c-4e9b-88c8-2af7a5871a62 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:32.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb499258-7e47-4fad-8196-29502a898776 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:32.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb499258-7e47-4fad-8196-29502a898776 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f5e5047 2021-05-04T00:12:32.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb499258-7e47-4fad-8196-29502a898776 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:32.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb499258-7e47-4fad-8196-29502a898776 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:12:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:12:35.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d51a0ba-fd58-4ca5-a87e-73e1c0dab3dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:35.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d51a0ba-fd58-4ca5-a87e-73e1c0dab3dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41ffa6a5 2021-05-04T00:12:35.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d51a0ba-fd58-4ca5-a87e-73e1c0dab3dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:35.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d51a0ba-fd58-4ca5-a87e-73e1c0dab3dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:37.108Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:12:37.109Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:12:37.109Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:12:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50052470-6fa4-416f-b4b4-451a1904e99c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50052470-6fa4-416f-b4b4-451a1904e99c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1634f950 2021-05-04T00:12:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50052470-6fa4-416f-b4b4-451a1904e99c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50052470-6fa4-416f-b4b4-451a1904e99c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:41.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b80c019-69d1-429d-8b84-f75ed78adf6e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:41.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b80c019-69d1-429d-8b84-f75ed78adf6e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f5357b7 2021-05-04T00:12:41.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b80c019-69d1-429d-8b84-f75ed78adf6e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:41.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b80c019-69d1-429d-8b84-f75ed78adf6e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b3b649-f9ef-4ace-800a-f8eee0b97bad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b3b649-f9ef-4ace-800a-f8eee0b97bad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62b2e872 2021-05-04T00:12:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b3b649-f9ef-4ace-800a-f8eee0b97bad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b3b649-f9ef-4ace-800a-f8eee0b97bad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:47.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4009ed69-68b7-4a49-aa06-2dd7048f44f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:47.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4009ed69-68b7-4a49-aa06-2dd7048f44f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e44a00f 2021-05-04T00:12:47.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4009ed69-68b7-4a49-aa06-2dd7048f44f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:47.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4009ed69-68b7-4a49-aa06-2dd7048f44f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:12:50.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:12:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f6bae4-9982-4050-ac20-795b0c06baa8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f6bae4-9982-4050-ac20-795b0c06baa8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f0b2cee 2021-05-04T00:12:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f6bae4-9982-4050-ac20-795b0c06baa8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f6bae4-9982-4050-ac20-795b0c06baa8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:52.066Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:12:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62100d10-f2ba-4210-9cfe-6fd4e5567b40 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62100d10-f2ba-4210-9cfe-6fd4e5567b40 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a203057 2021-05-04T00:12:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62100d10-f2ba-4210-9cfe-6fd4e5567b40 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62100d10-f2ba-4210-9cfe-6fd4e5567b40 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:56.546Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cac19be1-68e2-4ac7-a277-18a769c39d00 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:56.546Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cac19be1-68e2-4ac7-a277-18a769c39d00 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f24bc19 2021-05-04T00:12:56.546Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cac19be1-68e2-4ac7-a277-18a769c39d00 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:56.546Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cac19be1-68e2-4ac7-a277-18a769c39d00 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:12:59.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fda894a-e14f-4f03-8905-97c4a3876b2f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:12:59.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fda894a-e14f-4f03-8905-97c4a3876b2f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@241b3b2a 2021-05-04T00:12:59.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fda894a-e14f-4f03-8905-97c4a3876b2f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:12:59.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fda894a-e14f-4f03-8905-97c4a3876b2f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:02.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cc3cc4-eb11-43a8-a399-5680f85b81b7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:02.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cc3cc4-eb11-43a8-a399-5680f85b81b7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12c11f1b 2021-05-04T00:13:02.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cc3cc4-eb11-43a8-a399-5680f85b81b7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:02.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cc3cc4-eb11-43a8-a399-5680f85b81b7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:13:05.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:13:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4478e01-f878-452c-8d1f-1660d1f401b5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4478e01-f878-452c-8d1f-1660d1f401b5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3948379c 2021-05-04T00:13:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4478e01-f878-452c-8d1f-1660d1f401b5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4478e01-f878-452c-8d1f-1660d1f401b5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:08.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88455f71-d255-4d8c-9e20-61db49b01184 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:08.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88455f71-d255-4d8c-9e20-61db49b01184 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@437680d8 2021-05-04T00:13:08.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88455f71-d255-4d8c-9e20-61db49b01184 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:08.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88455f71-d255-4d8c-9e20-61db49b01184 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:11.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4792f51-b27b-46b9-a178-1a0541fe2473 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:11.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4792f51-b27b-46b9-a178-1a0541fe2473 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ec931f7 2021-05-04T00:13:11.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4792f51-b27b-46b9-a178-1a0541fe2473 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:11.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4792f51-b27b-46b9-a178-1a0541fe2473 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:14.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba0e82f1-9ed2-4cef-9a50-d66e9430f681 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:14.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba0e82f1-9ed2-4cef-9a50-d66e9430f681 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24ead8f2 2021-05-04T00:13:14.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba0e82f1-9ed2-4cef-9a50-d66e9430f681 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:14.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba0e82f1-9ed2-4cef-9a50-d66e9430f681 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae6b924e-8c96-46be-a469-b20dc910c198 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae6b924e-8c96-46be-a469-b20dc910c198 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c66ef24 2021-05-04T00:13:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae6b924e-8c96-46be-a469-b20dc910c198 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae6b924e-8c96-46be-a469-b20dc910c198 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:13:20.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:13:20.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff6aff54-373b-449e-91af-948f0fdbc344 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:20.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff6aff54-373b-449e-91af-948f0fdbc344 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b61c430 2021-05-04T00:13:20.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff6aff54-373b-449e-91af-948f0fdbc344 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:20.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff6aff54-373b-449e-91af-948f0fdbc344 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:23.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0712c4f-52a3-4e51-bebb-bb99443822c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:23.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0712c4f-52a3-4e51-bebb-bb99443822c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5cd01e22 2021-05-04T00:13:23.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0712c4f-52a3-4e51-bebb-bb99443822c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:23.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0712c4f-52a3-4e51-bebb-bb99443822c9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:25.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b020bab-a97d-4a70-8e8b-3c164f4561eb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:25.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b020bab-a97d-4a70-8e8b-3c164f4561eb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e0f7002 2021-05-04T00:13:25.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b020bab-a97d-4a70-8e8b-3c164f4561eb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:25.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b020bab-a97d-4a70-8e8b-3c164f4561eb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:29.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe6df03-237c-47a9-a48b-668b36dfbf80 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:29.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe6df03-237c-47a9-a48b-668b36dfbf80 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45fbb528 2021-05-04T00:13:29.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe6df03-237c-47a9-a48b-668b36dfbf80 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:29.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe6df03-237c-47a9-a48b-668b36dfbf80 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:32.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36341186-2f32-4749-9a7e-3cde16e83ac0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:32.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36341186-2f32-4749-9a7e-3cde16e83ac0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@601eb9e3 2021-05-04T00:13:32.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36341186-2f32-4749-9a7e-3cde16e83ac0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:32.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36341186-2f32-4749-9a7e-3cde16e83ac0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:35.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:13:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:13:35.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35f1a5ab-b74e-4381-96b8-b67f13eeef0c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:35.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35f1a5ab-b74e-4381-96b8-b67f13eeef0c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ce38acc 2021-05-04T00:13:35.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35f1a5ab-b74e-4381-96b8-b67f13eeef0c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:35.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35f1a5ab-b74e-4381-96b8-b67f13eeef0c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:37.108Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:13:37.109Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:13:37.109Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:13:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b72f49d-8ee6-413b-82c7-c2413691554b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b72f49d-8ee6-413b-82c7-c2413691554b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69b96232 2021-05-04T00:13:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b72f49d-8ee6-413b-82c7-c2413691554b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b72f49d-8ee6-413b-82c7-c2413691554b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:41.125Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5867d26-740b-4ca1-bf20-729e5c03cdf3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:41.125Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5867d26-740b-4ca1-bf20-729e5c03cdf3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7173d8d8 2021-05-04T00:13:41.125Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5867d26-740b-4ca1-bf20-729e5c03cdf3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:41.125Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5867d26-740b-4ca1-bf20-729e5c03cdf3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:42.219Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=c96a00b2-541c-4e6d-adba-0edabcea5cd8 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:13:42.218837Z InvocationID=f2a0905b-ba41-440f-b4f2-6f96e36ffad3 LogTimestamp=2021-05-04T00:13:42.218945Z Entering 2021-05-04T00:13:42.220Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=c96a00b2-541c-4e6d-adba-0edabcea5cd8 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:13:42.218837Z InvocationID=f2a0905b-ba41-440f-b4f2-6f96e36ffad3 LogTimestamp=2021-05-04T00:13:42.218945Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:13:42.223Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=2 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:13:42.220Z InvocationID=4d1d5c3d-7efe-4e50-8cc3-b2c46dd454e0 LogTimestamp=2021-05-04T00:13:42.222900Z ResponseDesc=OK Exiting. 2021-05-04T00:13:42.223Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:13:43.008Z [qtp580718781-11] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=f0d99c68-e70d-4e29-9b5d-41e09fd921f9 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:13:43.008310Z InvocationID=e3312924-1915-4012-985c-b084587454c5 LogTimestamp=2021-05-04T00:13:43.008378Z Entering 2021-05-04T00:13:43.009Z [qtp580718781-11] INFO o.o.sdc.be.filters.BeServletFilter RequestID=f0d99c68-e70d-4e29-9b5d-41e09fd921f9 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:13:43.008310Z InvocationID=e3312924-1915-4012-985c-b084587454c5 LogTimestamp=2021-05-04T00:13:43.008378Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:13:43.017Z [qtp580718781-11] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:13:43.009Z InvocationID=2133b569-bfe5-4ab5-9a4b-e467a36ac242 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/fOChk4fN1dJZcQcs will send credentials over a clear channel. 2021-05-04T00:13:43.017Z [qtp580718781-11] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:13:43.009Z InvocationID=2133b569-bfe5-4ab5-9a4b-e467a36ac242 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/fOChk4fN1dJZcQcs (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:13:43.043Z [qtp580718781-11] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:13:43.009Z InvocationID=2133b569-bfe5-4ab5-9a4b-e467a36ac242 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:13:43.051Z [qtp580718781-11] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "fOChk4fN1dJZcQcs" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:13:43.009Z InvocationID=2133b569-bfe5-4ab5-9a4b-e467a36ac242 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/fOChk4fN1dJZcQcs will send credentials over a clear channel. 2021-05-04T00:13:43.051Z [qtp580718781-11] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "fOChk4fN1dJZcQcs" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:13:43.009Z InvocationID=2133b569-bfe5-4ab5-9a4b-e467a36ac242 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/fOChk4fN1dJZcQcs (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:13:43.277Z [qtp580718781-11] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "fOChk4fN1dJZcQcs" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:13:43.009Z InvocationID=2133b569-bfe5-4ab5-9a4b-e467a36ac242 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:13:43.283Z [qtp580718781-11] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "cds" API_KEY = "fOChk4fN1dJZcQcs" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=274 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:13:43.009Z InvocationID=2133b569-bfe5-4ab5-9a4b-e467a36ac242 LogTimestamp=2021-05-04T00:13:43.283430Z ResponseDesc=OK Exiting. 2021-05-04T00:13:43.284Z [qtp580718781-11] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:13:44.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfadb26b-4a87-4cb2-9977-87022f7d75b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:44.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfadb26b-4a87-4cb2-9977-87022f7d75b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64b7a6b4 2021-05-04T00:13:44.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfadb26b-4a87-4cb2-9977-87022f7d75b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:44.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfadb26b-4a87-4cb2-9977-87022f7d75b9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:47.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f75a2aed-ca60-400a-bc27-49b02fed78c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:47.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f75a2aed-ca60-400a-bc27-49b02fed78c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4422f504 2021-05-04T00:13:47.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f75a2aed-ca60-400a-bc27-49b02fed78c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:47.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f75a2aed-ca60-400a-bc27-49b02fed78c8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:13:50.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:13:50.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f72aa8a0-354c-47cb-be1f-520fe1311406 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:50.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f72aa8a0-354c-47cb-be1f-520fe1311406 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66a8a740 2021-05-04T00:13:50.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f72aa8a0-354c-47cb-be1f-520fe1311406 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:50.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f72aa8a0-354c-47cb-be1f-520fe1311406 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:52.051Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:13:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c5213a2-ab23-4a62-a0ae-c76f80943b26 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c5213a2-ab23-4a62-a0ae-c76f80943b26 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9f20655 2021-05-04T00:13:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c5213a2-ab23-4a62-a0ae-c76f80943b26 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c5213a2-ab23-4a62-a0ae-c76f80943b26 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:56.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fda5488c-f5e4-4daa-a94a-ac3502086e38 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:56.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fda5488c-f5e4-4daa-a94a-ac3502086e38 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50a993d9 2021-05-04T00:13:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fda5488c-f5e4-4daa-a94a-ac3502086e38 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fda5488c-f5e4-4daa-a94a-ac3502086e38 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:13:59.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34dc7494-9b3e-48ec-82e4-e32d9f583dd5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:13:59.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34dc7494-9b3e-48ec-82e4-e32d9f583dd5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@370c61d 2021-05-04T00:13:59.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34dc7494-9b3e-48ec-82e4-e32d9f583dd5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:13:59.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34dc7494-9b3e-48ec-82e4-e32d9f583dd5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:02.532Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e445d7a-af34-4c8b-8875-50b5dbeef748 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:02.532Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e445d7a-af34-4c8b-8875-50b5dbeef748 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20c1f394 2021-05-04T00:14:02.532Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e445d7a-af34-4c8b-8875-50b5dbeef748 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:02.532Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e445d7a-af34-4c8b-8875-50b5dbeef748 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:14:05.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:14:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37a0d85e-9faa-4859-aef3-78f99220b8e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37a0d85e-9faa-4859-aef3-78f99220b8e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39189977 2021-05-04T00:14:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37a0d85e-9faa-4859-aef3-78f99220b8e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37a0d85e-9faa-4859-aef3-78f99220b8e1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:08.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a6b370d-4b6e-45c3-8615-57f561a19b5c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:08.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a6b370d-4b6e-45c3-8615-57f561a19b5c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2533f2d6 2021-05-04T00:14:08.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a6b370d-4b6e-45c3-8615-57f561a19b5c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:08.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a6b370d-4b6e-45c3-8615-57f561a19b5c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:11.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d2a5b3-a833-46d1-9846-c7874b76f590 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:11.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d2a5b3-a833-46d1-9846-c7874b76f590 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20f69843 2021-05-04T00:14:11.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d2a5b3-a833-46d1-9846-c7874b76f590 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:11.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d2a5b3-a833-46d1-9846-c7874b76f590 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b29668f-1aef-46f3-a2d7-72f35ae2a1b1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b29668f-1aef-46f3-a2d7-72f35ae2a1b1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f84829a 2021-05-04T00:14:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b29668f-1aef-46f3-a2d7-72f35ae2a1b1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b29668f-1aef-46f3-a2d7-72f35ae2a1b1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:17.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c9c3fca-132d-4058-8693-fff493c0c2f2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:17.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c9c3fca-132d-4058-8693-fff493c0c2f2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72aa613c 2021-05-04T00:14:17.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c9c3fca-132d-4058-8693-fff493c0c2f2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:17.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c9c3fca-132d-4058-8693-fff493c0c2f2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:20.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=765e6afb-c913-4d4e-9754-34ffc984b1c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:20.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=765e6afb-c913-4d4e-9754-34ffc984b1c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48b588fd 2021-05-04T00:14:20.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=765e6afb-c913-4d4e-9754-34ffc984b1c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:20.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=765e6afb-c913-4d4e-9754-34ffc984b1c8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:14:20.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:14:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7186af-1671-423c-9000-9abd6f738706 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7186af-1671-423c-9000-9abd6f738706 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c952528 2021-05-04T00:14:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7186af-1671-423c-9000-9abd6f738706 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7186af-1671-423c-9000-9abd6f738706 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:26.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c09bee85-b901-4cab-8d16-47d1d8d44c48 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:26.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c09bee85-b901-4cab-8d16-47d1d8d44c48 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77aefc72 2021-05-04T00:14:26.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c09bee85-b901-4cab-8d16-47d1d8d44c48 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:26.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c09bee85-b901-4cab-8d16-47d1d8d44c48 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:29.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13dc04ff-870b-4f64-a4ab-dcb5a866e9a1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13dc04ff-870b-4f64-a4ab-dcb5a866e9a1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@562cd03a 2021-05-04T00:14:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13dc04ff-870b-4f64-a4ab-dcb5a866e9a1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13dc04ff-870b-4f64-a4ab-dcb5a866e9a1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:32.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9347af7-93a6-4e88-af22-fc216849910c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:32.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9347af7-93a6-4e88-af22-fc216849910c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32ba4862 2021-05-04T00:14:32.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9347af7-93a6-4e88-af22-fc216849910c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:32.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9347af7-93a6-4e88-af22-fc216849910c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:14:35.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:14:35.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c93ece1-81bc-4e02-95d2-6dc3235826d2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:35.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c93ece1-81bc-4e02-95d2-6dc3235826d2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47739778 2021-05-04T00:14:35.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c93ece1-81bc-4e02-95d2-6dc3235826d2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:35.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c93ece1-81bc-4e02-95d2-6dc3235826d2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:14:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:14:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:14:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c560e2a-3e12-4a40-b858-d1b759169395 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c560e2a-3e12-4a40-b858-d1b759169395 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@473ebe88 2021-05-04T00:14:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c560e2a-3e12-4a40-b858-d1b759169395 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c560e2a-3e12-4a40-b858-d1b759169395 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:41.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a144b81-9abf-45d6-b9f2-28c8acfc616c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:41.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a144b81-9abf-45d6-b9f2-28c8acfc616c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77a31b55 2021-05-04T00:14:41.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a144b81-9abf-45d6-b9f2-28c8acfc616c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:41.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a144b81-9abf-45d6-b9f2-28c8acfc616c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:44.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7a35117-7206-4706-a13c-0e5c322270e2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:44.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7a35117-7206-4706-a13c-0e5c322270e2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17bab49b 2021-05-04T00:14:44.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7a35117-7206-4706-a13c-0e5c322270e2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:44.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7a35117-7206-4706-a13c-0e5c322270e2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:47.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=382dc62a-52ef-4536-a612-086c6599a177 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:47.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=382dc62a-52ef-4536-a612-086c6599a177 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ca93943 2021-05-04T00:14:47.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=382dc62a-52ef-4536-a612-086c6599a177 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:47.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=382dc62a-52ef-4536-a612-086c6599a177 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:14:50.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:14:50.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2ef90ef-ca82-4698-b580-54b5ce10105f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:50.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2ef90ef-ca82-4698-b580-54b5ce10105f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ed104a0 2021-05-04T00:14:50.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2ef90ef-ca82-4698-b580-54b5ce10105f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:50.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2ef90ef-ca82-4698-b580-54b5ce10105f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:52.041Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:14:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cde152aa-1854-4535-9ca3-714ef6f1e144 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cde152aa-1854-4535-9ca3-714ef6f1e144 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@11c7d19c 2021-05-04T00:14:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cde152aa-1854-4535-9ca3-714ef6f1e144 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cde152aa-1854-4535-9ca3-714ef6f1e144 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4454f90-3995-403f-8858-35e28520c2eb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4454f90-3995-403f-8858-35e28520c2eb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@539bb153 2021-05-04T00:14:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4454f90-3995-403f-8858-35e28520c2eb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4454f90-3995-403f-8858-35e28520c2eb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:14:59.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8308dece-66ca-4255-b2ec-e21b7b9220f2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:14:59.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8308dece-66ca-4255-b2ec-e21b7b9220f2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56fde942 2021-05-04T00:14:59.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8308dece-66ca-4255-b2ec-e21b7b9220f2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:14:59.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8308dece-66ca-4255-b2ec-e21b7b9220f2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:02.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56119853-f0bb-4fe5-832d-a487bdf707a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:02.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56119853-f0bb-4fe5-832d-a487bdf707a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@207f0339 2021-05-04T00:15:02.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56119853-f0bb-4fe5-832d-a487bdf707a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:02.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56119853-f0bb-4fe5-832d-a487bdf707a0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:05.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:15:05.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:15:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c79502f5-3a65-4f17-8d77-1a0de8cd7569 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c79502f5-3a65-4f17-8d77-1a0de8cd7569 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f9b6d9b 2021-05-04T00:15:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c79502f5-3a65-4f17-8d77-1a0de8cd7569 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c79502f5-3a65-4f17-8d77-1a0de8cd7569 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:08.596Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc9be1f7-41f6-4479-990b-03dd5918d4d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:08.596Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc9be1f7-41f6-4479-990b-03dd5918d4d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1845412 2021-05-04T00:15:08.597Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc9be1f7-41f6-4479-990b-03dd5918d4d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:08.597Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc9be1f7-41f6-4479-990b-03dd5918d4d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:11.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f9d8f1b-8547-49c3-a254-ab89c46caed8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:11.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f9d8f1b-8547-49c3-a254-ab89c46caed8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@380a1a99 2021-05-04T00:15:11.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f9d8f1b-8547-49c3-a254-ab89c46caed8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:11.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f9d8f1b-8547-49c3-a254-ab89c46caed8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba9225ec-62ef-4001-87dd-9224cb23bb27 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba9225ec-62ef-4001-87dd-9224cb23bb27 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6034b06a 2021-05-04T00:15:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba9225ec-62ef-4001-87dd-9224cb23bb27 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba9225ec-62ef-4001-87dd-9224cb23bb27 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:17.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1939029-d46f-49ea-9a8c-2c931f2cc58e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1939029-d46f-49ea-9a8c-2c931f2cc58e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3417fe42 2021-05-04T00:15:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1939029-d46f-49ea-9a8c-2c931f2cc58e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1939029-d46f-49ea-9a8c-2c931f2cc58e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:15:20.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:15:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27531d7d-752c-48f6-b706-63eeae0701db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27531d7d-752c-48f6-b706-63eeae0701db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17100816 2021-05-04T00:15:20.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27531d7d-752c-48f6-b706-63eeae0701db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:20.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27531d7d-752c-48f6-b706-63eeae0701db PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:23.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74c1ee79-6bd4-496c-aa29-a2d48aea23f3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:23.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74c1ee79-6bd4-496c-aa29-a2d48aea23f3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fa616bf 2021-05-04T00:15:23.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74c1ee79-6bd4-496c-aa29-a2d48aea23f3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:23.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74c1ee79-6bd4-496c-aa29-a2d48aea23f3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:26.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa22a760-5a86-44ec-8631-606040d71bd9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:26.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa22a760-5a86-44ec-8631-606040d71bd9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ad45ec1 2021-05-04T00:15:26.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa22a760-5a86-44ec-8631-606040d71bd9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:26.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa22a760-5a86-44ec-8631-606040d71bd9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:29.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=680c8bfb-10ca-42f1-81c3-db1912411284 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:29.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=680c8bfb-10ca-42f1-81c3-db1912411284 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@313a5237 2021-05-04T00:15:29.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=680c8bfb-10ca-42f1-81c3-db1912411284 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:29.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=680c8bfb-10ca-42f1-81c3-db1912411284 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:32.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a200e59b-8bef-4917-90c4-df38680f6709 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:32.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a200e59b-8bef-4917-90c4-df38680f6709 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@267dbcbe 2021-05-04T00:15:32.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a200e59b-8bef-4917-90c4-df38680f6709 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:32.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a200e59b-8bef-4917-90c4-df38680f6709 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:15:35.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:15:35.494Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ef61d5a-268b-4386-bd34-ba06d1e7407e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:35.494Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ef61d5a-268b-4386-bd34-ba06d1e7407e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69a14520 2021-05-04T00:15:35.494Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ef61d5a-268b-4386-bd34-ba06d1e7407e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:35.494Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ef61d5a-268b-4386-bd34-ba06d1e7407e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:15:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:15:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:15:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9670ab4-12df-4ea1-861c-61203afcfd61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9670ab4-12df-4ea1-861c-61203afcfd61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@711b2c83 2021-05-04T00:15:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9670ab4-12df-4ea1-861c-61203afcfd61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9670ab4-12df-4ea1-861c-61203afcfd61 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:42.376Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71bbe5e9-9372-44f2-997b-a69b22d1223d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:42.376Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71bbe5e9-9372-44f2-997b-a69b22d1223d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1253637d 2021-05-04T00:15:42.376Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71bbe5e9-9372-44f2-997b-a69b22d1223d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:42.376Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71bbe5e9-9372-44f2-997b-a69b22d1223d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:46.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=845bb74f-12f6-47e8-b622-a66bbc8f5f90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:46.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=845bb74f-12f6-47e8-b622-a66bbc8f5f90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5900a5c5 2021-05-04T00:15:46.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=845bb74f-12f6-47e8-b622-a66bbc8f5f90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:46.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=845bb74f-12f6-47e8-b622-a66bbc8f5f90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:47.649Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=321f7245-cebb-4dfd-a99e-6ce3c8f03806 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:47.649Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=321f7245-cebb-4dfd-a99e-6ce3c8f03806 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a6b6380 2021-05-04T00:15:47.649Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=321f7245-cebb-4dfd-a99e-6ce3c8f03806 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:47.649Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=321f7245-cebb-4dfd-a99e-6ce3c8f03806 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:15:50.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:15:50.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8756abf6-60e1-439b-a9a1-bce60e9ee91e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:50.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8756abf6-60e1-439b-a9a1-bce60e9ee91e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12bc3f7a 2021-05-04T00:15:50.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8756abf6-60e1-439b-a9a1-bce60e9ee91e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:50.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8756abf6-60e1-439b-a9a1-bce60e9ee91e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:52.041Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:15:53.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7060571-8f33-4d7a-8fd9-dd5d8afae27a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:53.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7060571-8f33-4d7a-8fd9-dd5d8afae27a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bfdfedb 2021-05-04T00:15:53.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7060571-8f33-4d7a-8fd9-dd5d8afae27a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:53.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7060571-8f33-4d7a-8fd9-dd5d8afae27a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:56.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9161988-3d9a-4be0-afa5-6273a9df748b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9161988-3d9a-4be0-afa5-6273a9df748b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@478f5eec 2021-05-04T00:15:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9161988-3d9a-4be0-afa5-6273a9df748b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9161988-3d9a-4be0-afa5-6273a9df748b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:15:59.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e81bb77-599e-4a32-bc3b-8dcb75fac5ad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:15:59.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e81bb77-599e-4a32-bc3b-8dcb75fac5ad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@261942fe 2021-05-04T00:15:59.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e81bb77-599e-4a32-bc3b-8dcb75fac5ad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:15:59.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e81bb77-599e-4a32-bc3b-8dcb75fac5ad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:02.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47b9117f-830f-4124-b895-1426dcd24e3d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:02.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47b9117f-830f-4124-b895-1426dcd24e3d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55adfea8 2021-05-04T00:16:02.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47b9117f-830f-4124-b895-1426dcd24e3d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:02.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47b9117f-830f-4124-b895-1426dcd24e3d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:16:05.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:16:05.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf215b68-1013-42b1-b4b3-ab0ed23495c5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:05.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf215b68-1013-42b1-b4b3-ab0ed23495c5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2152d9e0 2021-05-04T00:16:05.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf215b68-1013-42b1-b4b3-ab0ed23495c5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:05.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf215b68-1013-42b1-b4b3-ab0ed23495c5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:08.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c0f2f74-dda4-4904-97a7-7578d73c4dd5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:08.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c0f2f74-dda4-4904-97a7-7578d73c4dd5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c178b18 2021-05-04T00:16:08.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c0f2f74-dda4-4904-97a7-7578d73c4dd5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:08.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c0f2f74-dda4-4904-97a7-7578d73c4dd5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:11.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deef009e-557a-4c55-bdf0-c2557f743ed3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:11.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deef009e-557a-4c55-bdf0-c2557f743ed3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@707524d6 2021-05-04T00:16:11.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deef009e-557a-4c55-bdf0-c2557f743ed3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:11.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deef009e-557a-4c55-bdf0-c2557f743ed3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:14.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6c0b017-1e42-45fb-9735-2970027265a7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:14.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6c0b017-1e42-45fb-9735-2970027265a7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a17dcce 2021-05-04T00:16:14.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6c0b017-1e42-45fb-9735-2970027265a7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:14.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6c0b017-1e42-45fb-9735-2970027265a7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:17.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5a604af-dc3f-450a-b666-6d0312ff69e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:17.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5a604af-dc3f-450a-b666-6d0312ff69e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7dbd81b5 2021-05-04T00:16:17.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5a604af-dc3f-450a-b666-6d0312ff69e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:17.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5a604af-dc3f-450a-b666-6d0312ff69e5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:20.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:16:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:16:20.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe30fc5-fbdc-4227-ad7c-00c53997e20f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:20.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe30fc5-fbdc-4227-ad7c-00c53997e20f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5dd29a51 2021-05-04T00:16:20.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe30fc5-fbdc-4227-ad7c-00c53997e20f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:20.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe30fc5-fbdc-4227-ad7c-00c53997e20f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:23.409Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=3fe30fc5-fbdc-4227-ad7c-00c53997e20f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:16:23.410Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=3fe30fc5-fbdc-4227-ad7c-00c53997e20f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:16:23.410Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3fe30fc5-fbdc-4227-ad7c-00c53997e20f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:16:23.410Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3fe30fc5-fbdc-4227-ad7c-00c53997e20f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:16:24.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c308c24-625f-4842-8563-3d5952776f65 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:24.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c308c24-625f-4842-8563-3d5952776f65 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d846a9 2021-05-04T00:16:24.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c308c24-625f-4842-8563-3d5952776f65 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:24.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c308c24-625f-4842-8563-3d5952776f65 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:24.308Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5c308c24-625f-4842-8563-3d5952776f65 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:16:24.308Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5c308c24-625f-4842-8563-3d5952776f65 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:16:26.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a817a4a3-16ad-4e80-88ea-b403a0a7de1a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:26.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a817a4a3-16ad-4e80-88ea-b403a0a7de1a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61fc8cc7 2021-05-04T00:16:26.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a817a4a3-16ad-4e80-88ea-b403a0a7de1a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:26.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a817a4a3-16ad-4e80-88ea-b403a0a7de1a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:29.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3e03680-3fd2-4634-bea1-92a125f0f720 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:29.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3e03680-3fd2-4634-bea1-92a125f0f720 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28b582bb 2021-05-04T00:16:29.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3e03680-3fd2-4634-bea1-92a125f0f720 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:29.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3e03680-3fd2-4634-bea1-92a125f0f720 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:31.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1b367e0-323b-4820-b0c7-3cdd04596f28 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:31.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1b367e0-323b-4820-b0c7-3cdd04596f28 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e1e740a 2021-05-04T00:16:31.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1b367e0-323b-4820-b0c7-3cdd04596f28 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:31.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1b367e0-323b-4820-b0c7-3cdd04596f28 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:16:35.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=665d0676-3dfe-41a8-aceb-aeb92c2a0fa7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:35.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=665d0676-3dfe-41a8-aceb-aeb92c2a0fa7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7601e512 2021-05-04T00:16:35.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=665d0676-3dfe-41a8-aceb-aeb92c2a0fa7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:35.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=665d0676-3dfe-41a8-aceb-aeb92c2a0fa7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:16:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:16:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:16:39.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5c6b557-07bb-4518-bbdb-3eceb56f379f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:39.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5c6b557-07bb-4518-bbdb-3eceb56f379f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@739ed5a1 2021-05-04T00:16:39.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5c6b557-07bb-4518-bbdb-3eceb56f379f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:39.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5c6b557-07bb-4518-bbdb-3eceb56f379f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:41.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2272fe73-f1d8-49a1-8e0e-a47dd178bf6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:41.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2272fe73-f1d8-49a1-8e0e-a47dd178bf6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@773a3133 2021-05-04T00:16:41.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2272fe73-f1d8-49a1-8e0e-a47dd178bf6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:41.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2272fe73-f1d8-49a1-8e0e-a47dd178bf6a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:43.169Z [UEB-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend probably lost connectivity to UEB Cluster. Please check the logs for more information. 2021-05-04T00:16:44.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5973ff3-d1b7-4d9f-90d0-c8d983b23785 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:44.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5973ff3-d1b7-4d9f-90d0-c8d983b23785 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a99b5fa 2021-05-04T00:16:44.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5973ff3-d1b7-4d9f-90d0-c8d983b23785 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:44.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5973ff3-d1b7-4d9f-90d0-c8d983b23785 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:47.416Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d5973ff3-d1b7-4d9f-90d0-c8d983b23785 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:16:47.416Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d5973ff3-d1b7-4d9f-90d0-c8d983b23785 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:16:47.416Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d5973ff3-d1b7-4d9f-90d0-c8d983b23785 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:16:47.416Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d5973ff3-d1b7-4d9f-90d0-c8d983b23785 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:16:49.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3739b29e-1d5e-48e3-9b64-fea0704dcb6d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:49.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3739b29e-1d5e-48e3-9b64-fea0704dcb6d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1349f4ce 2021-05-04T00:16:49.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3739b29e-1d5e-48e3-9b64-fea0704dcb6d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:49.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3739b29e-1d5e-48e3-9b64-fea0704dcb6d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:49.505Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3739b29e-1d5e-48e3-9b64-fea0704dcb6d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:16:49.505Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3739b29e-1d5e-48e3-9b64-fea0704dcb6d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:16:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6f513f2-24db-45ce-9c23-13b398c965e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6f513f2-24db-45ce-9c23-13b398c965e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30ea008e 2021-05-04T00:16:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6f513f2-24db-45ce-9c23-13b398c965e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:50.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6f513f2-24db-45ce-9c23-13b398c965e7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97693086-6966-4f45-b0df-6461341e4252 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97693086-6966-4f45-b0df-6461341e4252 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17fb550c 2021-05-04T00:16:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97693086-6966-4f45-b0df-6461341e4252 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97693086-6966-4f45-b0df-6461341e4252 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:56.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2dab1b4b-e5d7-4b87-8337-b25add8468f7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:56.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2dab1b4b-e5d7-4b87-8337-b25add8468f7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b483cb9 2021-05-04T00:16:56.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2dab1b4b-e5d7-4b87-8337-b25add8468f7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:56.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2dab1b4b-e5d7-4b87-8337-b25add8468f7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:16:59.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f93c3f-ecd7-431f-986b-7d6ce8c81d05 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:16:59.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f93c3f-ecd7-431f-986b-7d6ce8c81d05 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@251f4ccb 2021-05-04T00:16:59.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f93c3f-ecd7-431f-986b-7d6ce8c81d05 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:16:59.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f93c3f-ecd7-431f-986b-7d6ce8c81d05 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:02.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f1ee01b-407e-469b-afe0-91a0bdec3973 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:02.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f1ee01b-407e-469b-afe0-91a0bdec3973 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4591c696 2021-05-04T00:17:02.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f1ee01b-407e-469b-afe0-91a0bdec3973 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:02.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f1ee01b-407e-469b-afe0-91a0bdec3973 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:05.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed3b6582-0fd6-48ca-bc09-b3408c71d1d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:05.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed3b6582-0fd6-48ca-bc09-b3408c71d1d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@80fb81d 2021-05-04T00:17:05.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed3b6582-0fd6-48ca-bc09-b3408c71d1d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:05.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed3b6582-0fd6-48ca-bc09-b3408c71d1d6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:08.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb7ad543-17da-4654-93ac-edec5241618e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:08.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb7ad543-17da-4654-93ac-edec5241618e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a8e051 2021-05-04T00:17:08.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb7ad543-17da-4654-93ac-edec5241618e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:08.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb7ad543-17da-4654-93ac-edec5241618e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:11.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bebd1bf-e361-4609-8924-f3cb45d06f93 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:11.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bebd1bf-e361-4609-8924-f3cb45d06f93 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d8e8ddb 2021-05-04T00:17:11.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bebd1bf-e361-4609-8924-f3cb45d06f93 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:11.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bebd1bf-e361-4609-8924-f3cb45d06f93 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:14.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bb4c373-4102-4ae1-a31c-65d08991601e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:14.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bb4c373-4102-4ae1-a31c-65d08991601e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1660558a 2021-05-04T00:17:14.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bb4c373-4102-4ae1-a31c-65d08991601e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:14.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bb4c373-4102-4ae1-a31c-65d08991601e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:17.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e677566-9113-423a-803d-2b4c9713440c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e677566-9113-423a-803d-2b4c9713440c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63dac80b 2021-05-04T00:17:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e677566-9113-423a-803d-2b4c9713440c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e677566-9113-423a-803d-2b4c9713440c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:20.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56c75504-2a7e-490e-b177-644f6d78d897 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:20.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56c75504-2a7e-490e-b177-644f6d78d897 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26350649 2021-05-04T00:17:20.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56c75504-2a7e-490e-b177-644f6d78d897 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:20.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56c75504-2a7e-490e-b177-644f6d78d897 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:23.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f008b29-943a-48e0-98df-890494f77859 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:23.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f008b29-943a-48e0-98df-890494f77859 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ffcb701 2021-05-04T00:17:23.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f008b29-943a-48e0-98df-890494f77859 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:23.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f008b29-943a-48e0-98df-890494f77859 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:25.988Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48fff79-64ad-4c59-883f-f11e7f6979e2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:25.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48fff79-64ad-4c59-883f-f11e7f6979e2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b247f71 2021-05-04T00:17:25.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48fff79-64ad-4c59-883f-f11e7f6979e2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:25.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48fff79-64ad-4c59-883f-f11e7f6979e2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:29.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69325f9b-d6d1-4e1c-9643-06752298894e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:29.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69325f9b-d6d1-4e1c-9643-06752298894e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e0b768e 2021-05-04T00:17:29.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69325f9b-d6d1-4e1c-9643-06752298894e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:29.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69325f9b-d6d1-4e1c-9643-06752298894e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:31.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3c82e24-b666-47d7-b01a-7925b57faef2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:31.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3c82e24-b666-47d7-b01a-7925b57faef2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20822a48 2021-05-04T00:17:31.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3c82e24-b666-47d7-b01a-7925b57faef2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:31.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3c82e24-b666-47d7-b01a-7925b57faef2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=545fcbac-93e3-4e5d-adf2-ee6b6786cbfd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=545fcbac-93e3-4e5d-adf2-ee6b6786cbfd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a2cd5e6 2021-05-04T00:17:35.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=545fcbac-93e3-4e5d-adf2-ee6b6786cbfd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:35.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=545fcbac-93e3-4e5d-adf2-ee6b6786cbfd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed1c5780-df59-4873-b56d-a09bfeef3220 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed1c5780-df59-4873-b56d-a09bfeef3220 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72cbe425 2021-05-04T00:17:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed1c5780-df59-4873-b56d-a09bfeef3220 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed1c5780-df59-4873-b56d-a09bfeef3220 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:41.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c726a1a4-f35a-40f6-9118-645e142c46e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:41.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c726a1a4-f35a-40f6-9118-645e142c46e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78143f4 2021-05-04T00:17:41.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c726a1a4-f35a-40f6-9118-645e142c46e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:41.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c726a1a4-f35a-40f6-9118-645e142c46e5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:44.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc47048a-2c57-4af2-8f8d-678f1c0c0ef7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:44.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc47048a-2c57-4af2-8f8d-678f1c0c0ef7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3660f926 2021-05-04T00:17:44.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc47048a-2c57-4af2-8f8d-678f1c0c0ef7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:44.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc47048a-2c57-4af2-8f8d-678f1c0c0ef7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:46.826Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e06feee5-13ab-46b1-ad8f-6fc45734d56d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:46.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e06feee5-13ab-46b1-ad8f-6fc45734d56d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a4f99e9 2021-05-04T00:17:46.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e06feee5-13ab-46b1-ad8f-6fc45734d56d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:46.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e06feee5-13ab-46b1-ad8f-6fc45734d56d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:50.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=922ba5fc-10ff-42ee-b582-99e38e9c1815 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:50.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=922ba5fc-10ff-42ee-b582-99e38e9c1815 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3708ac49 2021-05-04T00:17:50.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=922ba5fc-10ff-42ee-b582-99e38e9c1815 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:50.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=922ba5fc-10ff-42ee-b582-99e38e9c1815 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaae3a7f-e49b-42d2-b1ac-7db41cc1f682 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaae3a7f-e49b-42d2-b1ac-7db41cc1f682 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a9bd7bf 2021-05-04T00:17:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaae3a7f-e49b-42d2-b1ac-7db41cc1f682 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaae3a7f-e49b-42d2-b1ac-7db41cc1f682 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40a18644-219e-4439-acd3-28584e260d5c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:56.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40a18644-219e-4439-acd3-28584e260d5c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b03b003 2021-05-04T00:17:56.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40a18644-219e-4439-acd3-28584e260d5c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:56.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40a18644-219e-4439-acd3-28584e260d5c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:17:59.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b831a82-5ad0-421d-a739-b278e97400ee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:17:59.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b831a82-5ad0-421d-a739-b278e97400ee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d65e05b 2021-05-04T00:17:59.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b831a82-5ad0-421d-a739-b278e97400ee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:17:59.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b831a82-5ad0-421d-a739-b278e97400ee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:02.527Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26c980e6-28ff-4be1-88c3-f7255cbc7c86 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:02.527Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26c980e6-28ff-4be1-88c3-f7255cbc7c86 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f7d735a 2021-05-04T00:18:02.527Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26c980e6-28ff-4be1-88c3-f7255cbc7c86 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:02.527Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26c980e6-28ff-4be1-88c3-f7255cbc7c86 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9feb280f-2aa2-4f9f-8e43-d253c1fc1d96 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9feb280f-2aa2-4f9f-8e43-d253c1fc1d96 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7bbf6925 2021-05-04T00:18:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9feb280f-2aa2-4f9f-8e43-d253c1fc1d96 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9feb280f-2aa2-4f9f-8e43-d253c1fc1d96 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:08.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=649d7366-4fd8-4430-a4e7-11a3e3309ac1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=649d7366-4fd8-4430-a4e7-11a3e3309ac1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d2a4df9 2021-05-04T00:18:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=649d7366-4fd8-4430-a4e7-11a3e3309ac1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=649d7366-4fd8-4430-a4e7-11a3e3309ac1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:11.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d561b964-0140-49c6-a9d8-fae8f8dbd6cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:11.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d561b964-0140-49c6-a9d8-fae8f8dbd6cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f8f1f3f 2021-05-04T00:18:11.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d561b964-0140-49c6-a9d8-fae8f8dbd6cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:11.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d561b964-0140-49c6-a9d8-fae8f8dbd6cb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=090c6169-607c-4631-9d27-a2d9456fe173 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=090c6169-607c-4631-9d27-a2d9456fe173 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b1c4e37 2021-05-04T00:18:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=090c6169-607c-4631-9d27-a2d9456fe173 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=090c6169-607c-4631-9d27-a2d9456fe173 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:17.333Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2e005ef-7ef1-4af3-85d1-b44603b866f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:17.334Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2e005ef-7ef1-4af3-85d1-b44603b866f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b6ae230 2021-05-04T00:18:17.334Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2e005ef-7ef1-4af3-85d1-b44603b866f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:17.334Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2e005ef-7ef1-4af3-85d1-b44603b866f5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:20.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6da52331-ffdb-463b-accc-0c8667a75457 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:20.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6da52331-ffdb-463b-accc-0c8667a75457 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64102921 2021-05-04T00:18:20.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6da52331-ffdb-463b-accc-0c8667a75457 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:20.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6da52331-ffdb-463b-accc-0c8667a75457 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:23.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fed0bc39-1b85-4c6d-846b-94c33ca51f40 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:23.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fed0bc39-1b85-4c6d-846b-94c33ca51f40 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33b1b47b 2021-05-04T00:18:23.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fed0bc39-1b85-4c6d-846b-94c33ca51f40 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:23.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fed0bc39-1b85-4c6d-846b-94c33ca51f40 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:26.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c770772-9dd6-4f15-90be-8590cb8cba4b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:26.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c770772-9dd6-4f15-90be-8590cb8cba4b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ae539cc 2021-05-04T00:18:26.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c770772-9dd6-4f15-90be-8590cb8cba4b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:26.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c770772-9dd6-4f15-90be-8590cb8cba4b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:29.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33a93e98-dc6a-4fd0-893f-5f4cdf0c82e8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:29.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33a93e98-dc6a-4fd0-893f-5f4cdf0c82e8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69231c2 2021-05-04T00:18:29.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33a93e98-dc6a-4fd0-893f-5f4cdf0c82e8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:29.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33a93e98-dc6a-4fd0-893f-5f4cdf0c82e8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:31.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=171ab33f-2498-49c2-8195-3ae1288884d0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:31.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=171ab33f-2498-49c2-8195-3ae1288884d0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@632e5910 2021-05-04T00:18:31.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=171ab33f-2498-49c2-8195-3ae1288884d0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:31.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=171ab33f-2498-49c2-8195-3ae1288884d0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:35.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9d6a6e3-9be4-4902-b471-e963841a36b1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:35.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9d6a6e3-9be4-4902-b471-e963841a36b1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d022b62 2021-05-04T00:18:35.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9d6a6e3-9be4-4902-b471-e963841a36b1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:35.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9d6a6e3-9be4-4902-b471-e963841a36b1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:38.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f90a988-163f-4974-be21-f54d69670509 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:38.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f90a988-163f-4974-be21-f54d69670509 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e8a5257 2021-05-04T00:18:38.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f90a988-163f-4974-be21-f54d69670509 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:38.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f90a988-163f-4974-be21-f54d69670509 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:18:42.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:18:43.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8f0d3c1-96b9-4b74-a624-1e5c99cb477e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:43.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8f0d3c1-96b9-4b74-a624-1e5c99cb477e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69817065 2021-05-04T00:18:43.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8f0d3c1-96b9-4b74-a624-1e5c99cb477e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:43.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8f0d3c1-96b9-4b74-a624-1e5c99cb477e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:44.480Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e2bc8c8-8971-48ad-8c16-e61458d53314 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:44.480Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e2bc8c8-8971-48ad-8c16-e61458d53314 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7876529c 2021-05-04T00:18:44.480Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e2bc8c8-8971-48ad-8c16-e61458d53314 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:44.480Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e2bc8c8-8971-48ad-8c16-e61458d53314 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:45.156Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4e2bc8c8-8971-48ad-8c16-e61458d53314 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:18:45.157Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=4e2bc8c8-8971-48ad-8c16-e61458d53314 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:18:45.204Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:18:46.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bce0137f-d22d-4008-bf84-790ce2fda217 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:46.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bce0137f-d22d-4008-bf84-790ce2fda217 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34632fcb 2021-05-04T00:18:46.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bce0137f-d22d-4008-bf84-790ce2fda217 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:46.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bce0137f-d22d-4008-bf84-790ce2fda217 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:47.216Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:18:50.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6954947d-b8b8-4b31-8dec-4094d979f302 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:50.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6954947d-b8b8-4b31-8dec-4094d979f302 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18d7cbfa 2021-05-04T00:18:50.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6954947d-b8b8-4b31-8dec-4094d979f302 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:50.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6954947d-b8b8-4b31-8dec-4094d979f302 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:50.567Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6954947d-b8b8-4b31-8dec-4094d979f302 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:18:50.567Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6954947d-b8b8-4b31-8dec-4094d979f302 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:18:53.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36cd0213-ad8e-40b5-b450-6764aa3a77d3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:53.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36cd0213-ad8e-40b5-b450-6764aa3a77d3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a3792dd 2021-05-04T00:18:53.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36cd0213-ad8e-40b5-b450-6764aa3a77d3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:53.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36cd0213-ad8e-40b5-b450-6764aa3a77d3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:55.513Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.520Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.529Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.530Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.541Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.542Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.546Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.548Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.560Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.561Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.571Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.572Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.577Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.578Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.582Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.583Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.588Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.589Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.593Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.595Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.599Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.600Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.605Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.606Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.611Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.612Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.616Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.617Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.621Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.623Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.626Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.629Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.637Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.639Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.724Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.726Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.945Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:55.947Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:18:55.976Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 503 Service Unavailable 2021-05-04T00:18:55.979Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 503 2021-05-04T00:18:55.980Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=503, variables=[]] 2021-05-04T00:18:55.980Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:18:55.980Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:18:55.981Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:18:55.981Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:18:55.990Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 503 Service Unavailable 2021-05-04T00:18:55.990Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 503 2021-05-04T00:18:55.990Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=503, variables=[]] 2021-05-04T00:18:55.990Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:18:55.990Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:18:55.990Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:18:55.991Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:18:56.010Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:18:56.011Z [UEB-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend connection recovery to UEB Cluster. 2021-05-04T00:18:56.041Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 503 Service Unavailable 2021-05-04T00:18:56.042Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 503 2021-05-04T00:18:56.042Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=503, variables=[]] 2021-05-04T00:18:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a2b31b-3990-4a91-8e75-1847904e6870 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a2b31b-3990-4a91-8e75-1847904e6870 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@309fddb5 2021-05-04T00:18:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a2b31b-3990-4a91-8e75-1847904e6870 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a2b31b-3990-4a91-8e75-1847904e6870 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:58.366Z [qtp580718781-13] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=aaa9ce5f-4c76-46e1-bbff-b17b573eb595 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:18:58.365543Z InvocationID=969ca5f8-7c1f-402b-934f-b0583479a673 LogTimestamp=2021-05-04T00:18:58.365687Z Entering 2021-05-04T00:18:58.366Z [qtp580718781-13] INFO o.o.sdc.be.filters.BeServletFilter RequestID=aaa9ce5f-4c76-46e1-bbff-b17b573eb595 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:18:58.365543Z InvocationID=969ca5f8-7c1f-402b-934f-b0583479a673 LogTimestamp=2021-05-04T00:18:58.365687Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:18:58.371Z [qtp580718781-13] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=3 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:18:58.367Z InvocationID=313448e8-2990-47e5-8d05-a7eb94575ba1 LogTimestamp=2021-05-04T00:18:58.370921Z ResponseDesc=OK Exiting. 2021-05-04T00:18:58.371Z [qtp580718781-13] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:18:59.120Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=0cb20dcf-2755-4835-8bc3-ff2b471faa06 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:18:59.119733Z InvocationID=019b2e9c-8b6a-48cd-aa57-1b2966589ffb LogTimestamp=2021-05-04T00:18:59.119818Z Entering 2021-05-04T00:18:59.120Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=0cb20dcf-2755-4835-8bc3-ff2b471faa06 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:18:59.119733Z InvocationID=019b2e9c-8b6a-48cd-aa57-1b2966589ffb LogTimestamp=2021-05-04T00:18:59.119818Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:18:59.127Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:18:59.120Z InvocationID=a62b75fe-e42c-4dc6-9e81-2c12fa75ce38 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/cQnSfLzEPFSbycHo will send credentials over a clear channel. 2021-05-04T00:18:59.127Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:18:59.120Z InvocationID=a62b75fe-e42c-4dc6-9e81-2c12fa75ce38 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/cQnSfLzEPFSbycHo (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:18:59.346Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c262cfdc-09bc-4145-b693-f67b3b90c781 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:18:59.346Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c262cfdc-09bc-4145-b693-f67b3b90c781 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d2fa11a 2021-05-04T00:18:59.346Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c262cfdc-09bc-4145-b693-f67b3b90c781 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:18:59.346Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c262cfdc-09bc-4145-b693-f67b3b90c781 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:18:59.708Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c262cfdc-09bc-4145-b693-f67b3b90c781 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:18:59.708Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c262cfdc-09bc-4145-b693-f67b3b90c781 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:18:59.710Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:18:59.120Z InvocationID=a62b75fe-e42c-4dc6-9e81-2c12fa75ce38 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:18:59.718Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "cQnSfLzEPFSbycHo" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:18:59.120Z InvocationID=a62b75fe-e42c-4dc6-9e81-2c12fa75ce38 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/cQnSfLzEPFSbycHo will send credentials over a clear channel. 2021-05-04T00:18:59.719Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "cQnSfLzEPFSbycHo" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:18:59.120Z InvocationID=a62b75fe-e42c-4dc6-9e81-2c12fa75ce38 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/cQnSfLzEPFSbycHo (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:19:00.237Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "cQnSfLzEPFSbycHo" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:18:59.120Z InvocationID=a62b75fe-e42c-4dc6-9e81-2c12fa75ce38 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:19:00.350Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "ves-openapi-manager" API_KEY = "cQnSfLzEPFSbycHo" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=1230 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:18:59.120Z InvocationID=a62b75fe-e42c-4dc6-9e81-2c12fa75ce38 LogTimestamp=2021-05-04T00:19:00.350463Z ResponseDesc=OK Exiting. 2021-05-04T00:19:00.351Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:19:03.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f7a6348-2117-4887-8c6c-6a811e0ef240 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:03.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f7a6348-2117-4887-8c6c-6a811e0ef240 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b01315e 2021-05-04T00:19:03.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f7a6348-2117-4887-8c6c-6a811e0ef240 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:03.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f7a6348-2117-4887-8c6c-6a811e0ef240 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:19:05.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:19:05.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eca161ae-4de2-451c-8abf-5b0fe2d45ac6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:05.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eca161ae-4de2-451c-8abf-5b0fe2d45ac6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7de6fd99 2021-05-04T00:19:05.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eca161ae-4de2-451c-8abf-5b0fe2d45ac6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:05.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eca161ae-4de2-451c-8abf-5b0fe2d45ac6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:08.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=995cbb8c-3dac-4e03-8492-86a9eeafef13 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:08.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=995cbb8c-3dac-4e03-8492-86a9eeafef13 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33dae5e5 2021-05-04T00:19:08.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=995cbb8c-3dac-4e03-8492-86a9eeafef13 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:08.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=995cbb8c-3dac-4e03-8492-86a9eeafef13 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:11.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=835a126e-4c3e-4043-839f-e499a2b85eb2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:11.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=835a126e-4c3e-4043-839f-e499a2b85eb2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@539da5f5 2021-05-04T00:19:11.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=835a126e-4c3e-4043-839f-e499a2b85eb2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:11.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=835a126e-4c3e-4043-839f-e499a2b85eb2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:14.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3e86c37-8fa3-4191-ba5e-b6e05ffcba4f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:14.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3e86c37-8fa3-4191-ba5e-b6e05ffcba4f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61a4f519 2021-05-04T00:19:14.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3e86c37-8fa3-4191-ba5e-b6e05ffcba4f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:14.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3e86c37-8fa3-4191-ba5e-b6e05ffcba4f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:17.454Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0ec3108-fa52-4dd8-a856-5f7f599a3594 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:17.455Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0ec3108-fa52-4dd8-a856-5f7f599a3594 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b64c681 2021-05-04T00:19:17.455Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0ec3108-fa52-4dd8-a856-5f7f599a3594 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:17.455Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0ec3108-fa52-4dd8-a856-5f7f599a3594 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:19:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:19:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ef20801-bdb8-4c96-a411-f13b64c6be9f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ef20801-bdb8-4c96-a411-f13b64c6be9f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ff5ebc5 2021-05-04T00:19:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ef20801-bdb8-4c96-a411-f13b64c6be9f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ef20801-bdb8-4c96-a411-f13b64c6be9f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66f5791d-235d-47df-85be-f9362fa4bd55 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66f5791d-235d-47df-85be-f9362fa4bd55 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71794dc3 2021-05-04T00:19:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66f5791d-235d-47df-85be-f9362fa4bd55 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66f5791d-235d-47df-85be-f9362fa4bd55 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:26.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dc728de-70ba-4a14-b2ed-594530b51fe0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:26.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dc728de-70ba-4a14-b2ed-594530b51fe0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@373362c8 2021-05-04T00:19:26.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dc728de-70ba-4a14-b2ed-594530b51fe0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:26.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dc728de-70ba-4a14-b2ed-594530b51fe0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:29.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2590bf25-0f3c-44b8-a4dc-95cb84dfcc65 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:29.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2590bf25-0f3c-44b8-a4dc-95cb84dfcc65 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3aeeea61 2021-05-04T00:19:29.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2590bf25-0f3c-44b8-a4dc-95cb84dfcc65 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:29.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2590bf25-0f3c-44b8-a4dc-95cb84dfcc65 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:32.046Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7accc56e-17ea-41f2-804a-8575d4dbbfb1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:32.046Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7accc56e-17ea-41f2-804a-8575d4dbbfb1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a25853a 2021-05-04T00:19:32.046Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7accc56e-17ea-41f2-804a-8575d4dbbfb1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:32.046Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7accc56e-17ea-41f2-804a-8575d4dbbfb1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:19:35.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:19:35.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6bf166a-9894-4e60-ad4f-5169386dd980 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:35.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6bf166a-9894-4e60-ad4f-5169386dd980 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6109cce8 2021-05-04T00:19:35.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6bf166a-9894-4e60-ad4f-5169386dd980 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:35.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6bf166a-9894-4e60-ad4f-5169386dd980 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:19:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:19:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:19:38.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee58fd08-d4f8-40ed-96cf-eaf5a5e293d3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:38.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee58fd08-d4f8-40ed-96cf-eaf5a5e293d3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d00c723 2021-05-04T00:19:38.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee58fd08-d4f8-40ed-96cf-eaf5a5e293d3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:38.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee58fd08-d4f8-40ed-96cf-eaf5a5e293d3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:41.043Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6d7366c-5258-4d4a-85d2-98810b90edf5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:41.043Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6d7366c-5258-4d4a-85d2-98810b90edf5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f09100 2021-05-04T00:19:41.043Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6d7366c-5258-4d4a-85d2-98810b90edf5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:41.043Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6d7366c-5258-4d4a-85d2-98810b90edf5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:44.437Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9206fb18-6fb8-428f-b3cc-7502a865efdd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:44.437Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9206fb18-6fb8-428f-b3cc-7502a865efdd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@768ac871 2021-05-04T00:19:44.437Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9206fb18-6fb8-428f-b3cc-7502a865efdd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:44.437Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9206fb18-6fb8-428f-b3cc-7502a865efdd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:46.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc69331f-b93c-45c7-9254-738bcb46d1fb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:46.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc69331f-b93c-45c7-9254-738bcb46d1fb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59cb24c4 2021-05-04T00:19:46.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc69331f-b93c-45c7-9254-738bcb46d1fb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:46.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc69331f-b93c-45c7-9254-738bcb46d1fb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:19:50.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:19:50.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ab3ba9f-11ba-4a8d-b53a-670f23b25d7f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:50.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ab3ba9f-11ba-4a8d-b53a-670f23b25d7f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12b9c901 2021-05-04T00:19:50.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ab3ba9f-11ba-4a8d-b53a-670f23b25d7f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:50.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ab3ba9f-11ba-4a8d-b53a-670f23b25d7f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:53.168Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:19:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29690322-b67c-446e-aaaf-0b2365f4134b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29690322-b67c-446e-aaaf-0b2365f4134b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@164522c7 2021-05-04T00:19:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29690322-b67c-446e-aaaf-0b2365f4134b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29690322-b67c-446e-aaaf-0b2365f4134b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=598c7a12-39d8-4557-9701-172c874a7958 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=598c7a12-39d8-4557-9701-172c874a7958 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@36acb93e 2021-05-04T00:19:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=598c7a12-39d8-4557-9701-172c874a7958 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=598c7a12-39d8-4557-9701-172c874a7958 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:19:59.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5aa750b-3683-4e61-9d44-1127e9b08eda PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:19:59.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5aa750b-3683-4e61-9d44-1127e9b08eda PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3641462b 2021-05-04T00:19:59.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5aa750b-3683-4e61-9d44-1127e9b08eda PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:19:59.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5aa750b-3683-4e61-9d44-1127e9b08eda PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:02.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b56fbbb6-eb69-4621-911e-11d4726cfabf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:02.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b56fbbb6-eb69-4621-911e-11d4726cfabf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a6c6023 2021-05-04T00:20:02.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b56fbbb6-eb69-4621-911e-11d4726cfabf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:02.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b56fbbb6-eb69-4621-911e-11d4726cfabf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:20:05.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:20:05.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03903d36-a047-4d93-b6db-9b5917e2c19e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:05.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03903d36-a047-4d93-b6db-9b5917e2c19e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@558024b3 2021-05-04T00:20:05.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03903d36-a047-4d93-b6db-9b5917e2c19e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:05.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03903d36-a047-4d93-b6db-9b5917e2c19e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:08.061Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node015lg8qp9mxwzyxqcp6gmypw8w224 2021-05-04T00:20:08.061Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node084w2u6j6qlgf1rw2zwn3r3nsa232 2021-05-04T00:20:08.061Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node014dwnfhew3jh2vk7cdx561ozs213 2021-05-04T00:20:08.061Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01mtcgbnpkpb6rdm9dlynoken2229 2021-05-04T00:20:08.061Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node074tbowiaaah0khgimm3fp6i5231 2021-05-04T00:20:08.061Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0cmdj7u9cxsk11wfyp0mova0b3226 2021-05-04T00:20:08.062Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0he8s7c7vi94y1x2257zpv2x0s222 2021-05-04T00:20:08.062Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01abxrmjsnn6p31c9384b228ywo225 2021-05-04T00:20:08.062Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node09rthsy6xxxx2fw5okpoifxaq219 2021-05-04T00:20:08.062Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0q1465e23rx0ysneud7ma4l1h220 2021-05-04T00:20:08.062Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node04juqlukd6bworvdykl88jtqz228 2021-05-04T00:20:08.062Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01m02y7rsnvjtg1ozx902nwzpft227 2021-05-04T00:20:08.062Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0eqo57bw7z166nxonsyct8x96230 2021-05-04T00:20:08.062Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0k7b98gzvh5qnvq70v4bkwwpo233 2021-05-04T00:20:08.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9280f1df-1d96-44c4-a5c1-ebaffc942cbb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:08.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9280f1df-1d96-44c4-a5c1-ebaffc942cbb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6361274 2021-05-04T00:20:08.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9280f1df-1d96-44c4-a5c1-ebaffc942cbb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:08.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9280f1df-1d96-44c4-a5c1-ebaffc942cbb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:11.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec4fd319-64e5-499b-a2f9-49b46f048150 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:11.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec4fd319-64e5-499b-a2f9-49b46f048150 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3afa7c87 2021-05-04T00:20:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec4fd319-64e5-499b-a2f9-49b46f048150 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec4fd319-64e5-499b-a2f9-49b46f048150 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:14.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b80c202b-0a32-41e1-9726-f5a6cb772b30 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:14.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b80c202b-0a32-41e1-9726-f5a6cb772b30 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@339e7e83 2021-05-04T00:20:14.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b80c202b-0a32-41e1-9726-f5a6cb772b30 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:14.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b80c202b-0a32-41e1-9726-f5a6cb772b30 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:17.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf13e5a9-0189-4866-b145-ed5c71d52bdd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:17.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf13e5a9-0189-4866-b145-ed5c71d52bdd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67ef71ff 2021-05-04T00:20:17.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf13e5a9-0189-4866-b145-ed5c71d52bdd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:17.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf13e5a9-0189-4866-b145-ed5c71d52bdd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:20:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:20:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af0e6133-d9da-4024-baf3-078abf6c1cfb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af0e6133-d9da-4024-baf3-078abf6c1cfb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51fc6ac4 2021-05-04T00:20:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af0e6133-d9da-4024-baf3-078abf6c1cfb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af0e6133-d9da-4024-baf3-078abf6c1cfb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:23.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fd2eb2b-3cac-4edc-977d-6aaf7755030f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:23.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fd2eb2b-3cac-4edc-977d-6aaf7755030f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@388b2a38 2021-05-04T00:20:23.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fd2eb2b-3cac-4edc-977d-6aaf7755030f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:23.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fd2eb2b-3cac-4edc-977d-6aaf7755030f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:26.255Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8108285c-5c5d-4431-95ad-8357d784ec02 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:26.255Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8108285c-5c5d-4431-95ad-8357d784ec02 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78a131cc 2021-05-04T00:20:26.255Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8108285c-5c5d-4431-95ad-8357d784ec02 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:26.255Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8108285c-5c5d-4431-95ad-8357d784ec02 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:29.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fce505d3-e2f6-4150-b9e9-b9fe89c283fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:29.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fce505d3-e2f6-4150-b9e9-b9fe89c283fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64ddc7ae 2021-05-04T00:20:29.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fce505d3-e2f6-4150-b9e9-b9fe89c283fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:29.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fce505d3-e2f6-4150-b9e9-b9fe89c283fa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:32.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=949f4c0d-cbd8-4c78-bfeb-e5140f7909c5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:32.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=949f4c0d-cbd8-4c78-bfeb-e5140f7909c5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56453bbb 2021-05-04T00:20:32.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=949f4c0d-cbd8-4c78-bfeb-e5140f7909c5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:32.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=949f4c0d-cbd8-4c78-bfeb-e5140f7909c5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:20:35.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:20:35.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a9f0467-e4cb-45cb-bb9f-305f53a488a8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:35.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a9f0467-e4cb-45cb-bb9f-305f53a488a8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@377e11a1 2021-05-04T00:20:35.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a9f0467-e4cb-45cb-bb9f-305f53a488a8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:35.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a9f0467-e4cb-45cb-bb9f-305f53a488a8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:20:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:20:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:20:38.276Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71bf0668-d68d-45d5-9ea6-4b04761fd79f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:38.276Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71bf0668-d68d-45d5-9ea6-4b04761fd79f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46a069d3 2021-05-04T00:20:38.276Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71bf0668-d68d-45d5-9ea6-4b04761fd79f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:38.276Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71bf0668-d68d-45d5-9ea6-4b04761fd79f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:41.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8a94a86-56d3-4df0-b7db-1522029d4660 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:41.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8a94a86-56d3-4df0-b7db-1522029d4660 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c4a261e 2021-05-04T00:20:41.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8a94a86-56d3-4df0-b7db-1522029d4660 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:41.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8a94a86-56d3-4df0-b7db-1522029d4660 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9216345-6b10-40e1-a4ba-991ee81eeec3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9216345-6b10-40e1-a4ba-991ee81eeec3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@332d44cf 2021-05-04T00:20:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9216345-6b10-40e1-a4ba-991ee81eeec3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9216345-6b10-40e1-a4ba-991ee81eeec3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:46.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1ffca28-615b-4e0a-8055-ada073bbf7b2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:46.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1ffca28-615b-4e0a-8055-ada073bbf7b2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7419a812 2021-05-04T00:20:46.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1ffca28-615b-4e0a-8055-ada073bbf7b2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:46.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1ffca28-615b-4e0a-8055-ada073bbf7b2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:50.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:20:50.197Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:20:50.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74cb7e75-929c-4f91-af15-730b631ccfa7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:50.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74cb7e75-929c-4f91-af15-730b631ccfa7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3132a9b3 2021-05-04T00:20:50.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74cb7e75-929c-4f91-af15-730b631ccfa7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:50.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74cb7e75-929c-4f91-af15-730b631ccfa7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:52.062Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:20:53.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b9cfff0-7879-4aa1-9cab-7ce696b2fa13 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b9cfff0-7879-4aa1-9cab-7ce696b2fa13 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cd5035d 2021-05-04T00:20:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b9cfff0-7879-4aa1-9cab-7ce696b2fa13 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:53.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b9cfff0-7879-4aa1-9cab-7ce696b2fa13 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:56.752Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3ea6eff-101b-4d13-b2c7-bed48eb4286d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:56.752Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3ea6eff-101b-4d13-b2c7-bed48eb4286d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75872134 2021-05-04T00:20:56.753Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3ea6eff-101b-4d13-b2c7-bed48eb4286d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:56.753Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3ea6eff-101b-4d13-b2c7-bed48eb4286d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:20:59.481Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7b0e698-e8b1-4066-bad9-06011f387242 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:20:59.482Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7b0e698-e8b1-4066-bad9-06011f387242 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ce75eae 2021-05-04T00:20:59.482Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7b0e698-e8b1-4066-bad9-06011f387242 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:20:59.482Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7b0e698-e8b1-4066-bad9-06011f387242 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6b77a61-a238-4b5c-8b1e-9dda475d8dee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6b77a61-a238-4b5c-8b1e-9dda475d8dee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@683c09b0 2021-05-04T00:21:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6b77a61-a238-4b5c-8b1e-9dda475d8dee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6b77a61-a238-4b5c-8b1e-9dda475d8dee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:21:05.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:21:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e024cf5-7e80-4d13-a9ce-8086ea6b9d24 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e024cf5-7e80-4d13-a9ce-8086ea6b9d24 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6cf72001 2021-05-04T00:21:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e024cf5-7e80-4d13-a9ce-8086ea6b9d24 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e024cf5-7e80-4d13-a9ce-8086ea6b9d24 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:07.227Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=7eb07231-fec8-4e41-99c9-0bfb4c715a14 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:21:07.227051Z InvocationID=fc91e1c6-3e11-4596-8dc9-7f09f4edf08d LogTimestamp=2021-05-04T00:21:07.227233Z Entering 2021-05-04T00:21:07.228Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=7eb07231-fec8-4e41-99c9-0bfb4c715a14 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:21:07.227051Z InvocationID=fc91e1c6-3e11-4596-8dc9-7f09f4edf08d LogTimestamp=2021-05-04T00:21:07.227233Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:21:07.232Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:21:07.228Z InvocationID=79163b4b-8492-49a5-93c3-12d44001d5f0 LogTimestamp=2021-05-04T00:21:07.232280Z ResponseDesc=OK Exiting. 2021-05-04T00:21:07.232Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:21:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd8bb3ab-d63b-4272-9eb6-79dad0069dcd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd8bb3ab-d63b-4272-9eb6-79dad0069dcd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20b5222a 2021-05-04T00:21:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd8bb3ab-d63b-4272-9eb6-79dad0069dcd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:08.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd8bb3ab-d63b-4272-9eb6-79dad0069dcd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:11.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee8f1afa-84df-44ab-8864-815204b33807 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:11.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee8f1afa-84df-44ab-8864-815204b33807 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56510abc 2021-05-04T00:21:11.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee8f1afa-84df-44ab-8864-815204b33807 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:11.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee8f1afa-84df-44ab-8864-815204b33807 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:14.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c51e9f3-8f9b-47ef-b5e9-b03aa2cf1753 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:14.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c51e9f3-8f9b-47ef-b5e9-b03aa2cf1753 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ea8f84f 2021-05-04T00:21:14.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c51e9f3-8f9b-47ef-b5e9-b03aa2cf1753 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:14.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c51e9f3-8f9b-47ef-b5e9-b03aa2cf1753 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:16.615Z [qtp580718781-18] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=31beb631-dcd7-43e9-84ae-1d6439db4268 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:21:16.615533Z InvocationID=0d159c04-317b-46b5-aa70-aa55543654d2 LogTimestamp=2021-05-04T00:21:16.615592Z Entering 2021-05-04T00:21:16.615Z [qtp580718781-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=31beb631-dcd7-43e9-84ae-1d6439db4268 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:21:16.615533Z InvocationID=0d159c04-317b-46b5-aa70-aa55543654d2 LogTimestamp=2021-05-04T00:21:16.615592Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:21:16.617Z [qtp580718781-18] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:21:16.616Z InvocationID=d54f83f6-5aeb-480e-808d-f9d226601833 LogTimestamp=2021-05-04T00:21:16.617327Z ResponseDesc=OK Exiting. 2021-05-04T00:21:16.617Z [qtp580718781-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:21:16.854Z [qtp580718781-15] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=13477ce5-f041-457a-8300-a4b0f7befc3b InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:21:16.853759Z InvocationID=639115b5-17e3-4518-9950-3a3d1993220a LogTimestamp=2021-05-04T00:21:16.853833Z Entering 2021-05-04T00:21:16.854Z [qtp580718781-15] INFO o.o.sdc.be.filters.BeServletFilter RequestID=13477ce5-f041-457a-8300-a4b0f7befc3b InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:21:16.853759Z InvocationID=639115b5-17e3-4518-9950-3a3d1993220a LogTimestamp=2021-05-04T00:21:16.853833Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:21:16.860Z [qtp580718781-15] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:21:16.854Z InvocationID=4679d0e4-5dc4-4035-bfb3-5d9e9da0586f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/wWHSNPfxpJlvJE55 will send credentials over a clear channel. 2021-05-04T00:21:16.860Z [qtp580718781-15] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:21:16.854Z InvocationID=4679d0e4-5dc4-4035-bfb3-5d9e9da0586f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/wWHSNPfxpJlvJE55 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:21:16.882Z [qtp580718781-15] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:21:16.854Z InvocationID=4679d0e4-5dc4-4035-bfb3-5d9e9da0586f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:21:16.902Z [qtp580718781-15] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "wWHSNPfxpJlvJE55" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:21:16.854Z InvocationID=4679d0e4-5dc4-4035-bfb3-5d9e9da0586f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/wWHSNPfxpJlvJE55 will send credentials over a clear channel. 2021-05-04T00:21:16.902Z [qtp580718781-15] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "wWHSNPfxpJlvJE55" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:21:16.854Z InvocationID=4679d0e4-5dc4-4035-bfb3-5d9e9da0586f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/wWHSNPfxpJlvJE55 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:21:16.925Z [qtp580718781-15] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "wWHSNPfxpJlvJE55" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:21:16.854Z InvocationID=4679d0e4-5dc4-4035-bfb3-5d9e9da0586f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:21:17.003Z [qtp580718781-15] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "ves-openapi-manager" API_KEY = "wWHSNPfxpJlvJE55" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=149 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:21:16.854Z InvocationID=4679d0e4-5dc4-4035-bfb3-5d9e9da0586f LogTimestamp=2021-05-04T00:21:17.003538Z ResponseDesc=OK Exiting. 2021-05-04T00:21:17.003Z [qtp580718781-15] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:21:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b85cdb90-519d-475a-92b0-ca544ecc82dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b85cdb90-519d-475a-92b0-ca544ecc82dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b8a3cb3 2021-05-04T00:21:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b85cdb90-519d-475a-92b0-ca544ecc82dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b85cdb90-519d-475a-92b0-ca544ecc82dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:21:20.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:21:20.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78dd7019-952b-403e-b400-1d0515b83da0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:20.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78dd7019-952b-403e-b400-1d0515b83da0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@227eb9a 2021-05-04T00:21:20.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78dd7019-952b-403e-b400-1d0515b83da0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:20.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78dd7019-952b-403e-b400-1d0515b83da0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3513ef7-a1ae-4491-8c5a-510803c313df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3513ef7-a1ae-4491-8c5a-510803c313df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5cfd0588 2021-05-04T00:21:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3513ef7-a1ae-4491-8c5a-510803c313df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:23.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3513ef7-a1ae-4491-8c5a-510803c313df PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:26.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8e81c1-5208-41a2-9ffc-cdd448cede7c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:26.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8e81c1-5208-41a2-9ffc-cdd448cede7c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@450ef4d8 2021-05-04T00:21:26.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8e81c1-5208-41a2-9ffc-cdd448cede7c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:26.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8e81c1-5208-41a2-9ffc-cdd448cede7c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:29.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b60810c-6ee7-4f45-852e-0c222941bdda PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:29.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b60810c-6ee7-4f45-852e-0c222941bdda PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f3b376e 2021-05-04T00:21:29.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b60810c-6ee7-4f45-852e-0c222941bdda PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:29.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b60810c-6ee7-4f45-852e-0c222941bdda PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:32.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e30fd24-7b0e-4728-8ffb-484af73458c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:32.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e30fd24-7b0e-4728-8ffb-484af73458c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@227d7428 2021-05-04T00:21:32.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e30fd24-7b0e-4728-8ffb-484af73458c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:32.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e30fd24-7b0e-4728-8ffb-484af73458c9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:35.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:21:35.182Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:21:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e622767-6c99-40e2-af34-0044a9428fe6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e622767-6c99-40e2-af34-0044a9428fe6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1698e4aa 2021-05-04T00:21:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e622767-6c99-40e2-af34-0044a9428fe6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e622767-6c99-40e2-af34-0044a9428fe6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:21:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:21:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:21:38.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6241a10a-cb5b-409f-a6fe-ba8df023aede PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:38.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6241a10a-cb5b-409f-a6fe-ba8df023aede PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24027a95 2021-05-04T00:21:38.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6241a10a-cb5b-409f-a6fe-ba8df023aede PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:38.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6241a10a-cb5b-409f-a6fe-ba8df023aede PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:41.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c26ad5e-d9c5-44de-aed0-af94791b48ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:41.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c26ad5e-d9c5-44de-aed0-af94791b48ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6244b93c 2021-05-04T00:21:41.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c26ad5e-d9c5-44de-aed0-af94791b48ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:41.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c26ad5e-d9c5-44de-aed0-af94791b48ca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd5eda1b-8948-481c-a7a4-276ccd87e98f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd5eda1b-8948-481c-a7a4-276ccd87e98f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b514d68 2021-05-04T00:21:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd5eda1b-8948-481c-a7a4-276ccd87e98f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd5eda1b-8948-481c-a7a4-276ccd87e98f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:47.649Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b194848-0b6a-4519-83b7-4ec73672ef3b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:47.649Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b194848-0b6a-4519-83b7-4ec73672ef3b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a3d411c 2021-05-04T00:21:47.649Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b194848-0b6a-4519-83b7-4ec73672ef3b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:47.649Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b194848-0b6a-4519-83b7-4ec73672ef3b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:21:50.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:21:50.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=118737e5-59bf-4246-82f0-70ec5b72084a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:50.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=118737e5-59bf-4246-82f0-70ec5b72084a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ddc5a1c 2021-05-04T00:21:50.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=118737e5-59bf-4246-82f0-70ec5b72084a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:50.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=118737e5-59bf-4246-82f0-70ec5b72084a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:52.038Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:21:53.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=649cb8f3-263a-46db-9807-06b18f600ce2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:53.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=649cb8f3-263a-46db-9807-06b18f600ce2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a9697c2 2021-05-04T00:21:53.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=649cb8f3-263a-46db-9807-06b18f600ce2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:53.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=649cb8f3-263a-46db-9807-06b18f600ce2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:56.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c52b3fa-6c15-4819-bbc7-ce34cdc483d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:56.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c52b3fa-6c15-4819-bbc7-ce34cdc483d8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b0da1f5 2021-05-04T00:21:56.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c52b3fa-6c15-4819-bbc7-ce34cdc483d8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:56.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c52b3fa-6c15-4819-bbc7-ce34cdc483d8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:21:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee1aea5f-8a5d-4d8f-a2ab-460fe73de38c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:21:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee1aea5f-8a5d-4d8f-a2ab-460fe73de38c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b282029 2021-05-04T00:21:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee1aea5f-8a5d-4d8f-a2ab-460fe73de38c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:21:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee1aea5f-8a5d-4d8f-a2ab-460fe73de38c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:02.899Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47a27af0-3793-4b88-a043-77b6bbca1513 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:02.899Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47a27af0-3793-4b88-a043-77b6bbca1513 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@140b81ea 2021-05-04T00:22:02.899Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47a27af0-3793-4b88-a043-77b6bbca1513 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:02.899Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47a27af0-3793-4b88-a043-77b6bbca1513 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:22:05.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:22:05.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05049ce6-3be9-4d6a-b291-073de36d9d86 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:05.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05049ce6-3be9-4d6a-b291-073de36d9d86 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a9d3c2e 2021-05-04T00:22:05.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05049ce6-3be9-4d6a-b291-073de36d9d86 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:05.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05049ce6-3be9-4d6a-b291-073de36d9d86 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55677854-8658-4d12-bca7-800eb193ea92 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55677854-8658-4d12-bca7-800eb193ea92 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c593281 2021-05-04T00:22:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55677854-8658-4d12-bca7-800eb193ea92 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55677854-8658-4d12-bca7-800eb193ea92 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:11.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1f641b6-ac93-4eaf-9954-67157e058e81 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:11.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1f641b6-ac93-4eaf-9954-67157e058e81 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@168a6e4b 2021-05-04T00:22:11.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1f641b6-ac93-4eaf-9954-67157e058e81 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:11.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1f641b6-ac93-4eaf-9954-67157e058e81 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:14.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27c902f2-e838-4719-8f1a-df52183c29e2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:14.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27c902f2-e838-4719-8f1a-df52183c29e2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e46b9a1 2021-05-04T00:22:14.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27c902f2-e838-4719-8f1a-df52183c29e2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:14.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27c902f2-e838-4719-8f1a-df52183c29e2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:17.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c732c42-b5bf-47d6-9426-5fe85ed93906 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:17.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c732c42-b5bf-47d6-9426-5fe85ed93906 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1df9920b 2021-05-04T00:22:17.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c732c42-b5bf-47d6-9426-5fe85ed93906 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:17.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c732c42-b5bf-47d6-9426-5fe85ed93906 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:22:20.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e5b8e08-30a3-4a9c-b333-c1e3a350fa11 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:20.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e5b8e08-30a3-4a9c-b333-c1e3a350fa11 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3811660f 2021-05-04T00:22:20.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e5b8e08-30a3-4a9c-b333-c1e3a350fa11 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:20.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e5b8e08-30a3-4a9c-b333-c1e3a350fa11 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfd91ab9-daa8-4160-b5be-a63f52fcb8fb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfd91ab9-daa8-4160-b5be-a63f52fcb8fb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7dacaf70 2021-05-04T00:22:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfd91ab9-daa8-4160-b5be-a63f52fcb8fb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfd91ab9-daa8-4160-b5be-a63f52fcb8fb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:26.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e23cbd12-2fca-48c2-bdb0-b3e7800d0155 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:26.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e23cbd12-2fca-48c2-bdb0-b3e7800d0155 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16cc9ad5 2021-05-04T00:22:26.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e23cbd12-2fca-48c2-bdb0-b3e7800d0155 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:26.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e23cbd12-2fca-48c2-bdb0-b3e7800d0155 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:28.169Z [UEB-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend probably lost connectivity to UEB Cluster. Please check the logs for more information. 2021-05-04T00:22:29.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72d9373a-e23e-498f-bbc4-7fe95dea7a4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:29.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72d9373a-e23e-498f-bbc4-7fe95dea7a4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47742bcd 2021-05-04T00:22:29.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72d9373a-e23e-498f-bbc4-7fe95dea7a4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:29.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72d9373a-e23e-498f-bbc4-7fe95dea7a4e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:29.504Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=72d9373a-e23e-498f-bbc4-7fe95dea7a4e ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:22:29.504Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=72d9373a-e23e-498f-bbc4-7fe95dea7a4e ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:22:32.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7eee5385-eebb-4e6f-95f1-bf58c637ceff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:32.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7eee5385-eebb-4e6f-95f1-bf58c637ceff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74e61c32 2021-05-04T00:22:32.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7eee5385-eebb-4e6f-95f1-bf58c637ceff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:32.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7eee5385-eebb-4e6f-95f1-bf58c637ceff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04756be0-3ae7-4100-8890-81dc66880bfb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04756be0-3ae7-4100-8890-81dc66880bfb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c29faf1 2021-05-04T00:22:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04756be0-3ae7-4100-8890-81dc66880bfb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04756be0-3ae7-4100-8890-81dc66880bfb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:22:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:22:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:22:38.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6446143d-36e9-4981-9d3f-55fc69bb70f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:38.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6446143d-36e9-4981-9d3f-55fc69bb70f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ec1cf93 2021-05-04T00:22:38.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6446143d-36e9-4981-9d3f-55fc69bb70f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:38.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6446143d-36e9-4981-9d3f-55fc69bb70f5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:41.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eea0994-69e4-4687-b39c-ec3a52c7652d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:41.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eea0994-69e4-4687-b39c-ec3a52c7652d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@780da5c5 2021-05-04T00:22:41.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eea0994-69e4-4687-b39c-ec3a52c7652d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:41.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eea0994-69e4-4687-b39c-ec3a52c7652d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53e13491-90c2-468c-98c2-7497f194c562 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53e13491-90c2-468c-98c2-7497f194c562 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3cce317a 2021-05-04T00:22:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53e13491-90c2-468c-98c2-7497f194c562 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53e13491-90c2-468c-98c2-7497f194c562 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:47.138Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:22:47.142Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:22:47.148Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:22:47.149Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:22:47.153Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:22:47.155Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:22:47.159Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:22:47.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96fbe93f-6edb-4541-9613-4a5062814d4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:47.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96fbe93f-6edb-4541-9613-4a5062814d4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f93ba60 2021-05-04T00:22:47.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96fbe93f-6edb-4541-9613-4a5062814d4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:47.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96fbe93f-6edb-4541-9613-4a5062814d4a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:50.145Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e693ccd-3e5c-451b-a0ee-ab21e66522f3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:50.145Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e693ccd-3e5c-451b-a0ee-ab21e66522f3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f3e4356 2021-05-04T00:22:50.145Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e693ccd-3e5c-451b-a0ee-ab21e66522f3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:50.145Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e693ccd-3e5c-451b-a0ee-ab21e66522f3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:22:50.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:22:50.174Z [UEB-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend connection recovery to UEB Cluster. 2021-05-04T00:22:53.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87c29095-7d15-49c9-9251-d2d2e022737d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:53.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87c29095-7d15-49c9-9251-d2d2e022737d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a88fcf9 2021-05-04T00:22:53.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87c29095-7d15-49c9-9251-d2d2e022737d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:53.647Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87c29095-7d15-49c9-9251-d2d2e022737d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:54.057Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=87c29095-7d15-49c9-9251-d2d2e022737d ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:22:54.057Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=87c29095-7d15-49c9-9251-d2d2e022737d ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:22:56.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ec748e7-150f-471e-99ac-66f16e0976e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:56.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ec748e7-150f-471e-99ac-66f16e0976e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57805b33 2021-05-04T00:22:56.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ec748e7-150f-471e-99ac-66f16e0976e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:56.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ec748e7-150f-471e-99ac-66f16e0976e0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:22:59.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a044edf-b079-4511-acc8-85c3ec89352b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:22:59.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a044edf-b079-4511-acc8-85c3ec89352b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55c2ab19 2021-05-04T00:22:59.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a044edf-b079-4511-acc8-85c3ec89352b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:22:59.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a044edf-b079-4511-acc8-85c3ec89352b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:02.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98da0945-f5f9-44d0-947d-c2207ec22c44 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:02.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98da0945-f5f9-44d0-947d-c2207ec22c44 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@289b7604 2021-05-04T00:23:02.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98da0945-f5f9-44d0-947d-c2207ec22c44 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:02.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98da0945-f5f9-44d0-947d-c2207ec22c44 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:04.023Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:23:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:23:05.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:23:05.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2100d649-a77f-44de-bfb9-aac05da0b457 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:05.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2100d649-a77f-44de-bfb9-aac05da0b457 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75a928da 2021-05-04T00:23:05.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2100d649-a77f-44de-bfb9-aac05da0b457 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:05.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2100d649-a77f-44de-bfb9-aac05da0b457 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:08.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a96dd444-a010-48b5-8a84-a2bfc74aad31 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:08.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a96dd444-a010-48b5-8a84-a2bfc74aad31 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20a5bc9f 2021-05-04T00:23:08.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a96dd444-a010-48b5-8a84-a2bfc74aad31 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:08.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a96dd444-a010-48b5-8a84-a2bfc74aad31 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9b0e4be-494b-4d14-a51d-2e5a3f932ed8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9b0e4be-494b-4d14-a51d-2e5a3f932ed8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@607f41f1 2021-05-04T00:23:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9b0e4be-494b-4d14-a51d-2e5a3f932ed8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9b0e4be-494b-4d14-a51d-2e5a3f932ed8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:14.425Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f395439e-47f8-4acb-9799-df7ba5993cec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:14.425Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f395439e-47f8-4acb-9799-df7ba5993cec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13a730ac 2021-05-04T00:23:14.425Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f395439e-47f8-4acb-9799-df7ba5993cec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:14.425Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f395439e-47f8-4acb-9799-df7ba5993cec PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:17.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=215bbb82-bc30-47e2-986c-c2128fb4171a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:17.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=215bbb82-bc30-47e2-986c-c2128fb4171a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f7435fa 2021-05-04T00:23:17.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=215bbb82-bc30-47e2-986c-c2128fb4171a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:17.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=215bbb82-bc30-47e2-986c-c2128fb4171a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:23:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:23:20.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5d5293e-94b8-4348-b573-ba10c0a67528 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:20.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5d5293e-94b8-4348-b573-ba10c0a67528 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@223825cf 2021-05-04T00:23:20.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5d5293e-94b8-4348-b573-ba10c0a67528 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:20.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5d5293e-94b8-4348-b573-ba10c0a67528 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6549548d-2367-4b9f-a0b8-3c6d46d2877a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6549548d-2367-4b9f-a0b8-3c6d46d2877a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38d288b0 2021-05-04T00:23:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6549548d-2367-4b9f-a0b8-3c6d46d2877a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:23.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6549548d-2367-4b9f-a0b8-3c6d46d2877a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:26.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92720bfd-47c3-4718-b00a-ca827d5b7ea2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:26.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92720bfd-47c3-4718-b00a-ca827d5b7ea2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@125afd56 2021-05-04T00:23:26.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92720bfd-47c3-4718-b00a-ca827d5b7ea2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:26.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92720bfd-47c3-4718-b00a-ca827d5b7ea2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:29.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc6abf4f-848c-47f6-85b9-89b8567c7ab2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:29.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc6abf4f-848c-47f6-85b9-89b8567c7ab2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c8fd102 2021-05-04T00:23:29.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc6abf4f-848c-47f6-85b9-89b8567c7ab2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:29.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc6abf4f-848c-47f6-85b9-89b8567c7ab2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:32.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e937920e-c1e1-448f-8bab-de7c04879e7e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:32.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e937920e-c1e1-448f-8bab-de7c04879e7e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ffa8901 2021-05-04T00:23:32.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e937920e-c1e1-448f-8bab-de7c04879e7e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:32.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e937920e-c1e1-448f-8bab-de7c04879e7e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:23:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:23:35.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d4443ce-5dba-46f8-bf7f-55d70dfab7f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:35.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d4443ce-5dba-46f8-bf7f-55d70dfab7f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53c10946 2021-05-04T00:23:35.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d4443ce-5dba-46f8-bf7f-55d70dfab7f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:35.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d4443ce-5dba-46f8-bf7f-55d70dfab7f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:23:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:23:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:23:38.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfc02f7d-0edb-40bb-ac70-1092d8fee8e2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:38.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfc02f7d-0edb-40bb-ac70-1092d8fee8e2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1634fe25 2021-05-04T00:23:38.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfc02f7d-0edb-40bb-ac70-1092d8fee8e2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:38.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfc02f7d-0edb-40bb-ac70-1092d8fee8e2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:41.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aad44ae5-923f-4ef7-96c6-8bd062191bd4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:41.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aad44ae5-923f-4ef7-96c6-8bd062191bd4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74d13a20 2021-05-04T00:23:41.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aad44ae5-923f-4ef7-96c6-8bd062191bd4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:41.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aad44ae5-923f-4ef7-96c6-8bd062191bd4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:44.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=514c3165-1e0d-496b-b636-610f0fd6e971 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:44.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=514c3165-1e0d-496b-b636-610f0fd6e971 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f48173a 2021-05-04T00:23:44.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=514c3165-1e0d-496b-b636-610f0fd6e971 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:44.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=514c3165-1e0d-496b-b636-610f0fd6e971 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:46.994Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fba3e46-efeb-45bf-832f-af98d1ad7df8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:46.995Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fba3e46-efeb-45bf-832f-af98d1ad7df8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@498beac 2021-05-04T00:23:46.995Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fba3e46-efeb-45bf-832f-af98d1ad7df8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:46.995Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fba3e46-efeb-45bf-832f-af98d1ad7df8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:50.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f19a9ad0-aeda-4964-ac19-cb06f181f2fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:50.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f19a9ad0-aeda-4964-ac19-cb06f181f2fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35ecbdcc 2021-05-04T00:23:50.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f19a9ad0-aeda-4964-ac19-cb06f181f2fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:50.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f19a9ad0-aeda-4964-ac19-cb06f181f2fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:23:50.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:23:52.050Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:23:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=814a0b5f-8f79-4c72-b3d1-2b7c67ba277b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=814a0b5f-8f79-4c72-b3d1-2b7c67ba277b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4891816b 2021-05-04T00:23:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=814a0b5f-8f79-4c72-b3d1-2b7c67ba277b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=814a0b5f-8f79-4c72-b3d1-2b7c67ba277b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:56.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=078d9471-c362-4d73-b665-2fa052f2ffe3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:56.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=078d9471-c362-4d73-b665-2fa052f2ffe3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57ffbe84 2021-05-04T00:23:56.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=078d9471-c362-4d73-b665-2fa052f2ffe3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:56.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=078d9471-c362-4d73-b665-2fa052f2ffe3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:23:59.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc6bb68d-e795-4010-bd36-d545b29c4743 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:23:59.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc6bb68d-e795-4010-bd36-d545b29c4743 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16d31709 2021-05-04T00:23:59.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc6bb68d-e795-4010-bd36-d545b29c4743 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:23:59.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc6bb68d-e795-4010-bd36-d545b29c4743 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd0c7fa-a92f-4e63-b785-f81e7fac5ace PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:02.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd0c7fa-a92f-4e63-b785-f81e7fac5ace PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4681dac2 2021-05-04T00:24:02.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd0c7fa-a92f-4e63-b785-f81e7fac5ace PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:02.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd0c7fa-a92f-4e63-b785-f81e7fac5ace PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:24:05.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:24:05.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d86a3b23-b456-420b-a26b-8ca62f58218f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:05.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d86a3b23-b456-420b-a26b-8ca62f58218f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2df81a0 2021-05-04T00:24:05.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d86a3b23-b456-420b-a26b-8ca62f58218f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d86a3b23-b456-420b-a26b-8ca62f58218f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:08.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31f3b8f6-37cd-4cea-9d28-229eea413b74 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:08.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31f3b8f6-37cd-4cea-9d28-229eea413b74 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28fe14e1 2021-05-04T00:24:08.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31f3b8f6-37cd-4cea-9d28-229eea413b74 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:08.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31f3b8f6-37cd-4cea-9d28-229eea413b74 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6005043-8c6e-4452-9ebb-dd84066718ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6005043-8c6e-4452-9ebb-dd84066718ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@269b25a8 2021-05-04T00:24:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6005043-8c6e-4452-9ebb-dd84066718ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6005043-8c6e-4452-9ebb-dd84066718ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a5d80a3-d2d0-4088-af04-7d609fb6eb12 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a5d80a3-d2d0-4088-af04-7d609fb6eb12 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e04bc73 2021-05-04T00:24:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a5d80a3-d2d0-4088-af04-7d609fb6eb12 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a5d80a3-d2d0-4088-af04-7d609fb6eb12 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:17.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b5fb252-84f2-448d-9998-1e1235074d3a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:17.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b5fb252-84f2-448d-9998-1e1235074d3a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1028f292 2021-05-04T00:24:17.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b5fb252-84f2-448d-9998-1e1235074d3a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:17.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b5fb252-84f2-448d-9998-1e1235074d3a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:24:20.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:24:20.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f01753a5-2c6b-4db9-8573-5d6a74b56295 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:20.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f01753a5-2c6b-4db9-8573-5d6a74b56295 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48a9b56a 2021-05-04T00:24:20.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f01753a5-2c6b-4db9-8573-5d6a74b56295 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:20.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f01753a5-2c6b-4db9-8573-5d6a74b56295 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:23.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53ee9db4-6bab-4c63-9b6c-f1162c5ccfa8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:23.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53ee9db4-6bab-4c63-9b6c-f1162c5ccfa8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37e07fb2 2021-05-04T00:24:23.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53ee9db4-6bab-4c63-9b6c-f1162c5ccfa8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:23.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53ee9db4-6bab-4c63-9b6c-f1162c5ccfa8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:26.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28ab2d24-5540-488b-85c7-4f13de7cea0d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:26.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28ab2d24-5540-488b-85c7-4f13de7cea0d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cccba19 2021-05-04T00:24:26.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28ab2d24-5540-488b-85c7-4f13de7cea0d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:26.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28ab2d24-5540-488b-85c7-4f13de7cea0d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3795fd52-a409-4df8-b9f9-cd95646e89b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3795fd52-a409-4df8-b9f9-cd95646e89b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b277b70 2021-05-04T00:24:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3795fd52-a409-4df8-b9f9-cd95646e89b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3795fd52-a409-4df8-b9f9-cd95646e89b4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:32.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0630e671-9ea9-46ba-bcb8-c648be297c67 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:32.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0630e671-9ea9-46ba-bcb8-c648be297c67 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ea0899b 2021-05-04T00:24:32.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0630e671-9ea9-46ba-bcb8-c648be297c67 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:32.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0630e671-9ea9-46ba-bcb8-c648be297c67 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:24:35.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7a6f68a-6b7d-4b9a-81ac-2c496ae6aa0f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:35.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7a6f68a-6b7d-4b9a-81ac-2c496ae6aa0f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a897e12 2021-05-04T00:24:35.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7a6f68a-6b7d-4b9a-81ac-2c496ae6aa0f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:35.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7a6f68a-6b7d-4b9a-81ac-2c496ae6aa0f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:35.705Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:24:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:24:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:24:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:24:38.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54d38f0c-d778-4410-8afd-ef3c6f87a1c7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:38.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54d38f0c-d778-4410-8afd-ef3c6f87a1c7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30c1328 2021-05-04T00:24:38.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54d38f0c-d778-4410-8afd-ef3c6f87a1c7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:38.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54d38f0c-d778-4410-8afd-ef3c6f87a1c7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:41.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcd1b1fc-175a-4d53-925b-ffdc5bd29681 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:41.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcd1b1fc-175a-4d53-925b-ffdc5bd29681 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@136004c2 2021-05-04T00:24:41.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcd1b1fc-175a-4d53-925b-ffdc5bd29681 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:41.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcd1b1fc-175a-4d53-925b-ffdc5bd29681 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:44.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a27c28a-aab8-4651-abd0-a732df850846 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:44.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a27c28a-aab8-4651-abd0-a732df850846 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39a6bb64 2021-05-04T00:24:44.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a27c28a-aab8-4651-abd0-a732df850846 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:44.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a27c28a-aab8-4651-abd0-a732df850846 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:47.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fb1a1c4-eb17-40ff-808d-e31347887032 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:47.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fb1a1c4-eb17-40ff-808d-e31347887032 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@688e2bac 2021-05-04T00:24:47.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fb1a1c4-eb17-40ff-808d-e31347887032 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:47.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fb1a1c4-eb17-40ff-808d-e31347887032 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:24:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f390f6a-60ea-4e1d-944b-f5e64eb3dc5e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f390f6a-60ea-4e1d-944b-f5e64eb3dc5e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bf8e0c6 2021-05-04T00:24:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f390f6a-60ea-4e1d-944b-f5e64eb3dc5e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f390f6a-60ea-4e1d-944b-f5e64eb3dc5e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:53.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbf619de-0d84-46bd-996a-c8cafecdf7ee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:53.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbf619de-0d84-46bd-996a-c8cafecdf7ee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18ada36f 2021-05-04T00:24:53.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbf619de-0d84-46bd-996a-c8cafecdf7ee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:53.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbf619de-0d84-46bd-996a-c8cafecdf7ee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:54.275Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:24:56.331Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9efd3e8-c56b-4c6d-b08a-c96f0606bdfe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:56.331Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9efd3e8-c56b-4c6d-b08a-c96f0606bdfe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14dc5685 2021-05-04T00:24:56.331Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9efd3e8-c56b-4c6d-b08a-c96f0606bdfe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:56.331Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9efd3e8-c56b-4c6d-b08a-c96f0606bdfe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:58.168Z [UEB-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend probably lost connectivity to UEB Cluster. Please check the logs for more information. 2021-05-04T00:24:59.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da8a00df-006b-4c1c-ad7f-d3d2bd4f912b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:24:59.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da8a00df-006b-4c1c-ad7f-d3d2bd4f912b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d16dcf9 2021-05-04T00:24:59.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da8a00df-006b-4c1c-ad7f-d3d2bd4f912b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:24:59.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da8a00df-006b-4c1c-ad7f-d3d2bd4f912b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:24:59.223Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=da8a00df-006b-4c1c-ad7f-d3d2bd4f912b ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:24:59.224Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=da8a00df-006b-4c1c-ad7f-d3d2bd4f912b ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:25:02.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e03e080-536f-4233-b0ce-7d3958a6269d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:02.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e03e080-536f-4233-b0ce-7d3958a6269d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41a4e34d 2021-05-04T00:25:02.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e03e080-536f-4233-b0ce-7d3958a6269d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:02.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e03e080-536f-4233-b0ce-7d3958a6269d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:05.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=983ad6fc-0d0a-4fc9-857a-2e4b4ff5e2f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:05.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=983ad6fc-0d0a-4fc9-857a-2e4b4ff5e2f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34e796cd 2021-05-04T00:25:05.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=983ad6fc-0d0a-4fc9-857a-2e4b4ff5e2f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:05.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=983ad6fc-0d0a-4fc9-857a-2e4b4ff5e2f6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:08.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7728771-7d1a-44d9-9b92-c8dea39ffc28 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:08.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7728771-7d1a-44d9-9b92-c8dea39ffc28 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42f986aa 2021-05-04T00:25:08.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7728771-7d1a-44d9-9b92-c8dea39ffc28 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:08.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7728771-7d1a-44d9-9b92-c8dea39ffc28 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c35b250e-1225-450f-a7ab-4c7820540ee3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c35b250e-1225-450f-a7ab-4c7820540ee3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79336b18 2021-05-04T00:25:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c35b250e-1225-450f-a7ab-4c7820540ee3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:11.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c35b250e-1225-450f-a7ab-4c7820540ee3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:14.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5baea1d2-a66c-4e27-94ad-904a3ce08fba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:14.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5baea1d2-a66c-4e27-94ad-904a3ce08fba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f6b51f8 2021-05-04T00:25:14.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5baea1d2-a66c-4e27-94ad-904a3ce08fba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:14.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5baea1d2-a66c-4e27-94ad-904a3ce08fba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f76de99-09f8-4f2e-aced-8433ff55e1d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:17.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f76de99-09f8-4f2e-aced-8433ff55e1d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1732ed26 2021-05-04T00:25:17.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f76de99-09f8-4f2e-aced-8433ff55e1d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:17.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f76de99-09f8-4f2e-aced-8433ff55e1d7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:20.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57495814-6848-429c-94e0-19334a8b5802 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:20.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57495814-6848-429c-94e0-19334a8b5802 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1970e5bc 2021-05-04T00:25:20.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57495814-6848-429c-94e0-19334a8b5802 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:20.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57495814-6848-429c-94e0-19334a8b5802 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:23.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22dd53c5-7070-4387-88f9-787019bc601c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:23.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22dd53c5-7070-4387-88f9-787019bc601c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7bf92621 2021-05-04T00:25:23.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22dd53c5-7070-4387-88f9-787019bc601c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:23.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22dd53c5-7070-4387-88f9-787019bc601c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:26.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5601e24f-2646-41ab-9d6b-40d2efaa7e6b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:26.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5601e24f-2646-41ab-9d6b-40d2efaa7e6b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ed84c37 2021-05-04T00:25:26.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5601e24f-2646-41ab-9d6b-40d2efaa7e6b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:26.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5601e24f-2646-41ab-9d6b-40d2efaa7e6b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:29.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=522494d2-4c33-433e-bb39-3ff5bb9a75c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:29.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=522494d2-4c33-433e-bb39-3ff5bb9a75c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e83cd17 2021-05-04T00:25:29.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=522494d2-4c33-433e-bb39-3ff5bb9a75c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:29.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=522494d2-4c33-433e-bb39-3ff5bb9a75c8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=644c783b-b62d-481e-b3f8-87183b2c6c0a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=644c783b-b62d-481e-b3f8-87183b2c6c0a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2040927e 2021-05-04T00:25:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=644c783b-b62d-481e-b3f8-87183b2c6c0a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=644c783b-b62d-481e-b3f8-87183b2c6c0a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:35.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89b749c8-f393-40d9-83ef-dc0920c3d82a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:35.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89b749c8-f393-40d9-83ef-dc0920c3d82a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12ba6079 2021-05-04T00:25:35.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89b749c8-f393-40d9-83ef-dc0920c3d82a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:35.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89b749c8-f393-40d9-83ef-dc0920c3d82a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:25:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:25:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:25:38.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b8c9542-6a52-4ef8-9f94-8178ad498ae3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:38.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b8c9542-6a52-4ef8-9f94-8178ad498ae3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6eff3fed 2021-05-04T00:25:38.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b8c9542-6a52-4ef8-9f94-8178ad498ae3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:38.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b8c9542-6a52-4ef8-9f94-8178ad498ae3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:41.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46046bd4-5e1e-4060-bc86-651a74387b06 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:41.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46046bd4-5e1e-4060-bc86-651a74387b06 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e34ac50 2021-05-04T00:25:41.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46046bd4-5e1e-4060-bc86-651a74387b06 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:41.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46046bd4-5e1e-4060-bc86-651a74387b06 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:43.406Z [qtp580718781-19] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=f8190fbf-f82c-4d89-9337-c57abe6aaf32 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:25:43.405484Z InvocationID=729109d4-b42d-439a-9769-02fda2b9d852 LogTimestamp=2021-05-04T00:25:43.405703Z Entering 2021-05-04T00:25:43.406Z [qtp580718781-19] INFO o.o.sdc.be.filters.BeServletFilter RequestID=f8190fbf-f82c-4d89-9337-c57abe6aaf32 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:25:43.405484Z InvocationID=729109d4-b42d-439a-9769-02fda2b9d852 LogTimestamp=2021-05-04T00:25:43.405703Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:25:43.411Z [qtp580718781-19] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:25:43.407Z InvocationID=1471d3cb-b4c7-4bf6-a03f-ed40ca02b4ca LogTimestamp=2021-05-04T00:25:43.411422Z ResponseDesc=OK Exiting. 2021-05-04T00:25:43.412Z [qtp580718781-19] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:25:44.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=07550342-ac99-46ca-83a5-ce485d7eb975 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:44.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=07550342-ac99-46ca-83a5-ce485d7eb975 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a43148a 2021-05-04T00:25:44.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=07550342-ac99-46ca-83a5-ce485d7eb975 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:44.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=07550342-ac99-46ca-83a5-ce485d7eb975 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:46.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=294c6187-17d8-4a48-a4c2-888553d2fa1e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:46.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=294c6187-17d8-4a48-a4c2-888553d2fa1e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2bbb1813 2021-05-04T00:25:46.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=294c6187-17d8-4a48-a4c2-888553d2fa1e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:46.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=294c6187-17d8-4a48-a4c2-888553d2fa1e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:50.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62856834-c839-41a1-94e6-fb6e7be44445 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:50.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62856834-c839-41a1-94e6-fb6e7be44445 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c2c5ade 2021-05-04T00:25:50.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62856834-c839-41a1-94e6-fb6e7be44445 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:50.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62856834-c839-41a1-94e6-fb6e7be44445 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:53.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7c88f8f-f090-48dc-b2e4-2d0f4a5c337e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:53.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7c88f8f-f090-48dc-b2e4-2d0f4a5c337e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57912727 2021-05-04T00:25:53.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7c88f8f-f090-48dc-b2e4-2d0f4a5c337e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:53.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7c88f8f-f090-48dc-b2e4-2d0f4a5c337e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:56.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b46e2203-bb49-4dac-ad7c-fedc93317ae2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:56.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b46e2203-bb49-4dac-ad7c-fedc93317ae2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25849e1b 2021-05-04T00:25:56.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b46e2203-bb49-4dac-ad7c-fedc93317ae2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:56.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b46e2203-bb49-4dac-ad7c-fedc93317ae2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:25:59.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc7b65ea-4e1a-4dd0-bd27-27c583705abf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:25:59.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc7b65ea-4e1a-4dd0-bd27-27c583705abf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@568c6138 2021-05-04T00:25:59.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc7b65ea-4e1a-4dd0-bd27-27c583705abf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:25:59.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc7b65ea-4e1a-4dd0-bd27-27c583705abf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:02.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e32c5234-eb1c-40ef-bf17-88fbb7a342b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:02.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e32c5234-eb1c-40ef-bf17-88fbb7a342b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8283b9d 2021-05-04T00:26:02.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e32c5234-eb1c-40ef-bf17-88fbb7a342b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:02.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e32c5234-eb1c-40ef-bf17-88fbb7a342b9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:05.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=133774ff-f574-4458-8084-798f04ea0b03 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:05.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=133774ff-f574-4458-8084-798f04ea0b03 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56de8da0 2021-05-04T00:26:05.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=133774ff-f574-4458-8084-798f04ea0b03 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:05.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=133774ff-f574-4458-8084-798f04ea0b03 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4375ebeb-2fea-4bd1-a77e-12a0b34b856e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4375ebeb-2fea-4bd1-a77e-12a0b34b856e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57abb526 2021-05-04T00:26:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4375ebeb-2fea-4bd1-a77e-12a0b34b856e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4375ebeb-2fea-4bd1-a77e-12a0b34b856e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:12.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c33a0ffe-ac4c-44e5-93b1-65d53218c4bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:12.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c33a0ffe-ac4c-44e5-93b1-65d53218c4bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@621b9b4a 2021-05-04T00:26:12.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c33a0ffe-ac4c-44e5-93b1-65d53218c4bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:12.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c33a0ffe-ac4c-44e5-93b1-65d53218c4bd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:14.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7dfc0ea4-ae51-4b05-8fbc-cc1f6640e1a5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:14.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7dfc0ea4-ae51-4b05-8fbc-cc1f6640e1a5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ba6b047 2021-05-04T00:26:14.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7dfc0ea4-ae51-4b05-8fbc-cc1f6640e1a5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:14.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7dfc0ea4-ae51-4b05-8fbc-cc1f6640e1a5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:17.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dba8b01-5c7b-4a97-b2bb-583dfa526886 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:17.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dba8b01-5c7b-4a97-b2bb-583dfa526886 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9b78b1e 2021-05-04T00:26:17.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dba8b01-5c7b-4a97-b2bb-583dfa526886 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:17.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dba8b01-5c7b-4a97-b2bb-583dfa526886 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:20.276Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a9795f4-c5ac-4e70-a4da-13dedecc73e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:20.276Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a9795f4-c5ac-4e70-a4da-13dedecc73e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3bf1d8b 2021-05-04T00:26:20.276Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a9795f4-c5ac-4e70-a4da-13dedecc73e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:20.276Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a9795f4-c5ac-4e70-a4da-13dedecc73e1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:23.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8102c966-4033-4f8a-81f4-67f89af4bb41 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:23.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8102c966-4033-4f8a-81f4-67f89af4bb41 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3dc0077d 2021-05-04T00:26:23.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8102c966-4033-4f8a-81f4-67f89af4bb41 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:23.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8102c966-4033-4f8a-81f4-67f89af4bb41 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:26.877Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=731828ef-7c83-4f8e-b9c3-76d9fad09e2b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:26.877Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=731828ef-7c83-4f8e-b9c3-76d9fad09e2b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23faac0a 2021-05-04T00:26:26.877Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=731828ef-7c83-4f8e-b9c3-76d9fad09e2b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:26.877Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=731828ef-7c83-4f8e-b9c3-76d9fad09e2b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:29.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e73f3294-ea13-48a7-a9f9-2155542f84f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:29.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e73f3294-ea13-48a7-a9f9-2155542f84f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16203c10 2021-05-04T00:26:29.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e73f3294-ea13-48a7-a9f9-2155542f84f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:29.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e73f3294-ea13-48a7-a9f9-2155542f84f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=824acbb5-4c3b-4e4c-929b-89480d3806bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=824acbb5-4c3b-4e4c-929b-89480d3806bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d08281a 2021-05-04T00:26:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=824acbb5-4c3b-4e4c-929b-89480d3806bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:32.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=824acbb5-4c3b-4e4c-929b-89480d3806bd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:35.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e7d321b-5c90-4ec1-bce9-7e32beafa6a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:35.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e7d321b-5c90-4ec1-bce9-7e32beafa6a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50ce3129 2021-05-04T00:26:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e7d321b-5c90-4ec1-bce9-7e32beafa6a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e7d321b-5c90-4ec1-bce9-7e32beafa6a2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:38.329Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7198fab-d036-4ec0-844c-1b681d8a1c7a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:38.329Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7198fab-d036-4ec0-844c-1b681d8a1c7a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26a25dc6 2021-05-04T00:26:38.329Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7198fab-d036-4ec0-844c-1b681d8a1c7a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:38.329Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7198fab-d036-4ec0-844c-1b681d8a1c7a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:41.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7a1ba2b-3a7c-4aa3-bd18-a4bd29778c13 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:41.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7a1ba2b-3a7c-4aa3-bd18-a4bd29778c13 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@317535be 2021-05-04T00:26:41.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7a1ba2b-3a7c-4aa3-bd18-a4bd29778c13 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:41.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7a1ba2b-3a7c-4aa3-bd18-a4bd29778c13 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:44.445Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7ea6a44-8eff-413f-a91d-730eab5f63d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:44.445Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7ea6a44-8eff-413f-a91d-730eab5f63d8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d119372 2021-05-04T00:26:44.445Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7ea6a44-8eff-413f-a91d-730eab5f63d8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:44.445Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7ea6a44-8eff-413f-a91d-730eab5f63d8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:46.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=769c2da2-52fb-4a95-912a-31a7e2e81941 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:46.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=769c2da2-52fb-4a95-912a-31a7e2e81941 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c77be80 2021-05-04T00:26:46.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=769c2da2-52fb-4a95-912a-31a7e2e81941 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:46.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=769c2da2-52fb-4a95-912a-31a7e2e81941 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:50.136Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18e22032-a359-4e03-ae73-2e60cb4503eb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:50.137Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18e22032-a359-4e03-ae73-2e60cb4503eb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@88f88a5 2021-05-04T00:26:50.137Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18e22032-a359-4e03-ae73-2e60cb4503eb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:50.137Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18e22032-a359-4e03-ae73-2e60cb4503eb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:53.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6bd4c3a-cb78-4bd9-bafc-10017edc5ec5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:53.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6bd4c3a-cb78-4bd9-bafc-10017edc5ec5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@104f94cf 2021-05-04T00:26:53.802Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6bd4c3a-cb78-4bd9-bafc-10017edc5ec5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:53.802Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6bd4c3a-cb78-4bd9-bafc-10017edc5ec5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:56.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95b7c939-75bf-4d49-96b6-8a29a35b7043 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:56.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95b7c939-75bf-4d49-96b6-8a29a35b7043 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d41fb7 2021-05-04T00:26:56.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95b7c939-75bf-4d49-96b6-8a29a35b7043 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:56.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95b7c939-75bf-4d49-96b6-8a29a35b7043 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:26:59.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a1ae64c-c5d5-44ac-94d0-4fc2bfa05acf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:26:59.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a1ae64c-c5d5-44ac-94d0-4fc2bfa05acf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5622b3 2021-05-04T00:26:59.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a1ae64c-c5d5-44ac-94d0-4fc2bfa05acf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:26:59.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a1ae64c-c5d5-44ac-94d0-4fc2bfa05acf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:02.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c01285eb-e52d-4719-9ac5-3c699d9abacb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:02.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c01285eb-e52d-4719-9ac5-3c699d9abacb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74e848bc 2021-05-04T00:27:02.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c01285eb-e52d-4719-9ac5-3c699d9abacb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:02.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c01285eb-e52d-4719-9ac5-3c699d9abacb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:05.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd022bf5-28be-4ab9-8605-5819335b314c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:05.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd022bf5-28be-4ab9-8605-5819335b314c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@186d4302 2021-05-04T00:27:05.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd022bf5-28be-4ab9-8605-5819335b314c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:05.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd022bf5-28be-4ab9-8605-5819335b314c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:08.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e89931fe-b326-4723-943b-1d930edf9422 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:08.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e89931fe-b326-4723-943b-1d930edf9422 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39e5ce1f 2021-05-04T00:27:08.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e89931fe-b326-4723-943b-1d930edf9422 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:08.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e89931fe-b326-4723-943b-1d930edf9422 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:11.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9217a890-703c-40eb-b7d4-25ff10ff299b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:11.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9217a890-703c-40eb-b7d4-25ff10ff299b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@817202a 2021-05-04T00:27:11.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9217a890-703c-40eb-b7d4-25ff10ff299b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:11.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9217a890-703c-40eb-b7d4-25ff10ff299b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8b4bfd1-519c-4a0e-91d8-ca558e8a291f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8b4bfd1-519c-4a0e-91d8-ca558e8a291f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e8c243c 2021-05-04T00:27:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8b4bfd1-519c-4a0e-91d8-ca558e8a291f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8b4bfd1-519c-4a0e-91d8-ca558e8a291f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:17.435Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3258fd3-50fc-4441-8e19-0394d215c039 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:17.436Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3258fd3-50fc-4441-8e19-0394d215c039 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c8b05bc 2021-05-04T00:27:17.436Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3258fd3-50fc-4441-8e19-0394d215c039 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:17.436Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3258fd3-50fc-4441-8e19-0394d215c039 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=036af040-7cf3-40b9-881b-f307dcd20a90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=036af040-7cf3-40b9-881b-f307dcd20a90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b89af04 2021-05-04T00:27:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=036af040-7cf3-40b9-881b-f307dcd20a90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=036af040-7cf3-40b9-881b-f307dcd20a90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:22.691Z [qtp580718781-18] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=133c83db-d388-43aa-9ebc-09d74fd69b37 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:27:22.690245Z InvocationID=ca0e1202-6a3c-4040-ba11-c0aee504ea20 LogTimestamp=2021-05-04T00:27:22.690395Z Entering 2021-05-04T00:27:22.691Z [qtp580718781-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=133c83db-d388-43aa-9ebc-09d74fd69b37 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:27:22.690245Z InvocationID=ca0e1202-6a3c-4040-ba11-c0aee504ea20 LogTimestamp=2021-05-04T00:27:22.690395Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:27:22.694Z [qtp580718781-18] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=3 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:27:22.691Z InvocationID=f0ef0d65-f38a-4dfe-bdba-4a235eb512da LogTimestamp=2021-05-04T00:27:22.694536Z ResponseDesc=OK Exiting. 2021-05-04T00:27:22.695Z [qtp580718781-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:27:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=137ad4c2-b16b-4c05-bec5-c511ac8884f2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=137ad4c2-b16b-4c05-bec5-c511ac8884f2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a2ed22f 2021-05-04T00:27:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=137ad4c2-b16b-4c05-bec5-c511ac8884f2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=137ad4c2-b16b-4c05-bec5-c511ac8884f2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:25.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d230d261-e0e8-416f-83e4-7a9e538d02b7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:25.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d230d261-e0e8-416f-83e4-7a9e538d02b7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35dcb692 2021-05-04T00:27:25.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d230d261-e0e8-416f-83e4-7a9e538d02b7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:25.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d230d261-e0e8-416f-83e4-7a9e538d02b7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:29.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7e5e23-bdf2-4750-9d9a-e749d5dd0fd5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:29.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7e5e23-bdf2-4750-9d9a-e749d5dd0fd5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@216ef370 2021-05-04T00:27:29.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7e5e23-bdf2-4750-9d9a-e749d5dd0fd5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:29.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7e5e23-bdf2-4750-9d9a-e749d5dd0fd5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:32.152Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b0e678f-3ff3-4e17-97b1-6af8650aff6d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:32.152Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b0e678f-3ff3-4e17-97b1-6af8650aff6d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@370989d6 2021-05-04T00:27:32.152Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b0e678f-3ff3-4e17-97b1-6af8650aff6d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:32.152Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b0e678f-3ff3-4e17-97b1-6af8650aff6d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:35.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc630258-012b-4c01-8c75-008eb2491110 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:35.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc630258-012b-4c01-8c75-008eb2491110 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2baa8ab2 2021-05-04T00:27:35.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc630258-012b-4c01-8c75-008eb2491110 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:35.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc630258-012b-4c01-8c75-008eb2491110 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:38.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654813e9-3be2-4880-b294-7fad3232b25c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:38.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654813e9-3be2-4880-b294-7fad3232b25c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b12d173 2021-05-04T00:27:38.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654813e9-3be2-4880-b294-7fad3232b25c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:38.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654813e9-3be2-4880-b294-7fad3232b25c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:41.006Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f34f238-eeb5-476e-886f-7812f3322840 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:41.006Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f34f238-eeb5-476e-886f-7812f3322840 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2db15812 2021-05-04T00:27:41.006Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f34f238-eeb5-476e-886f-7812f3322840 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:41.006Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f34f238-eeb5-476e-886f-7812f3322840 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:43.936Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:43.943Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.039Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=871b1c77-7366-4317-a26f-a2a35f908f9e InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:27:44.039172Z InvocationID=9c541349-0528-4609-96b5-d2c78f5102c2 LogTimestamp=2021-05-04T00:27:44.039297Z Entering 2021-05-04T00:27:44.040Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=871b1c77-7366-4317-a26f-a2a35f908f9e InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:27:44.039172Z InvocationID=9c541349-0528-4609-96b5-d2c78f5102c2 LogTimestamp=2021-05-04T00:27:44.039297Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:27:44.046Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:27:44.040Z InvocationID=d1b2e5fe-0e1e-4d65-9515-de226c0cf177 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/KA1BErx6f2DirojN will send credentials over a clear channel. 2021-05-04T00:27:44.047Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:27:44.040Z InvocationID=d1b2e5fe-0e1e-4d65-9515-de226c0cf177 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/KA1BErx6f2DirojN (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:27:44.085Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.087Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.118Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.120Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.140Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.140Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.148Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.149Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.153Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.154Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.168Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.188Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.188Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.203Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.205Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.212Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.213Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.224Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.225Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.239Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.240Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.264Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.265Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.282Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.283Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.311Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.312Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09761744-be4b-445d-afb1-72278bd4eda8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:44.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09761744-be4b-445d-afb1-72278bd4eda8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@560d234b 2021-05-04T00:27:44.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09761744-be4b-445d-afb1-72278bd4eda8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:44.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09761744-be4b-445d-afb1-72278bd4eda8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:44.448Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:27:44.040Z InvocationID=d1b2e5fe-0e1e-4d65-9515-de226c0cf177 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.457Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "KA1BErx6f2DirojN" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:27:44.040Z InvocationID=d1b2e5fe-0e1e-4d65-9515-de226c0cf177 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/KA1BErx6f2DirojN will send credentials over a clear channel. 2021-05-04T00:27:44.457Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "KA1BErx6f2DirojN" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:27:44.040Z InvocationID=d1b2e5fe-0e1e-4d65-9515-de226c0cf177 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/KA1BErx6f2DirojN (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:27:44.489Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 503 Service Unavailable 2021-05-04T00:27:44.490Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 503 2021-05-04T00:27:44.491Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=503, variables=[]] 2021-05-04T00:27:44.491Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:27:44.491Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:27:44.491Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:27:44.491Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:27:44.530Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 503 Service Unavailable 2021-05-04T00:27:44.531Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 503 2021-05-04T00:27:44.531Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=503, variables=[]] 2021-05-04T00:27:44.531Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:27:44.531Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:27:44.531Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:27:44.531Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:27:44.550Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "KA1BErx6f2DirojN" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:27:44.040Z InvocationID=d1b2e5fe-0e1e-4d65-9515-de226c0cf177 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.558Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "ves-openapi-manager" API_KEY = "KA1BErx6f2DirojN" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=518 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:27:44.040Z InvocationID=d1b2e5fe-0e1e-4d65-9515-de226c0cf177 LogTimestamp=2021-05-04T00:27:44.558244Z ResponseDesc=OK Exiting. 2021-05-04T00:27:44.558Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:27:44.674Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.675Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.731Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.733Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.773Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 503 Service Unavailable 2021-05-04T00:27:44.773Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 503 2021-05-04T00:27:44.773Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=503, variables=[]] 2021-05-04T00:27:44.787Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.788Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.852Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.854Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.867Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.868Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.874Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.875Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.904Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.905Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:44.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:44.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:46.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef9a06ad-615e-46d5-a058-7414ff464422 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:46.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef9a06ad-615e-46d5-a058-7414ff464422 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@290ddc22 2021-05-04T00:27:46.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef9a06ad-615e-46d5-a058-7414ff464422 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:46.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef9a06ad-615e-46d5-a058-7414ff464422 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:27:50.211Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:27:50.211Z [UEB-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend connection recovery to UEB Cluster. 2021-05-04T00:27:50.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=368b21eb-92a8-45a6-90fe-9df4062bd2d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:50.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=368b21eb-92a8-45a6-90fe-9df4062bd2d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44790939 2021-05-04T00:27:50.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=368b21eb-92a8-45a6-90fe-9df4062bd2d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:50.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=368b21eb-92a8-45a6-90fe-9df4062bd2d7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:53.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89429574-74fd-4433-b2b3-fb580d5c4da7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:53.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89429574-74fd-4433-b2b3-fb580d5c4da7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63afa86c 2021-05-04T00:27:53.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89429574-74fd-4433-b2b3-fb580d5c4da7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:53.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89429574-74fd-4433-b2b3-fb580d5c4da7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:53.951Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=89429574-74fd-4433-b2b3-fb580d5c4da7 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:27:53.951Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=89429574-74fd-4433-b2b3-fb580d5c4da7 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:27:56.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5399480-d481-4204-bfed-d09e872a88b2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:56.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5399480-d481-4204-bfed-d09e872a88b2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c07c829 2021-05-04T00:27:56.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5399480-d481-4204-bfed-d09e872a88b2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:56.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5399480-d481-4204-bfed-d09e872a88b2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:27:59.554Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f5566c9-04f4-499a-b83f-d6206d854d81 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:27:59.554Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f5566c9-04f4-499a-b83f-d6206d854d81 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bacf347 2021-05-04T00:27:59.554Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f5566c9-04f4-499a-b83f-d6206d854d81 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:27:59.554Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f5566c9-04f4-499a-b83f-d6206d854d81 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:02.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=562079f3-6256-409d-894a-4a5a70663611 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:02.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=562079f3-6256-409d-894a-4a5a70663611 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@179011ca 2021-05-04T00:28:02.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=562079f3-6256-409d-894a-4a5a70663611 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:02.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=562079f3-6256-409d-894a-4a5a70663611 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:28:05.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:28:05.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cafd8fbb-5d48-4f00-b9e4-4a0574dfe8c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cafd8fbb-5d48-4f00-b9e4-4a0574dfe8c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@962279c 2021-05-04T00:28:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cafd8fbb-5d48-4f00-b9e4-4a0574dfe8c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cafd8fbb-5d48-4f00-b9e4-4a0574dfe8c9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:08.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db0070fd-eb07-4f41-86d0-837895ee5dc4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:08.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db0070fd-eb07-4f41-86d0-837895ee5dc4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6bc6f019 2021-05-04T00:28:08.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db0070fd-eb07-4f41-86d0-837895ee5dc4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:08.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db0070fd-eb07-4f41-86d0-837895ee5dc4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:11.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e3d880-49f7-4b48-88cf-a7ab2e85ba29 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:11.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e3d880-49f7-4b48-88cf-a7ab2e85ba29 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@170c00ed 2021-05-04T00:28:11.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e3d880-49f7-4b48-88cf-a7ab2e85ba29 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:11.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e3d880-49f7-4b48-88cf-a7ab2e85ba29 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:14.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d31219b-03ac-494c-a302-39474a14f891 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:14.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d31219b-03ac-494c-a302-39474a14f891 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54d87 2021-05-04T00:28:14.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d31219b-03ac-494c-a302-39474a14f891 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:14.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d31219b-03ac-494c-a302-39474a14f891 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f59c1a0-8f99-4e2c-b7ec-66be7f44252e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f59c1a0-8f99-4e2c-b7ec-66be7f44252e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18c3c4dc 2021-05-04T00:28:17.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f59c1a0-8f99-4e2c-b7ec-66be7f44252e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:17.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f59c1a0-8f99-4e2c-b7ec-66be7f44252e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:28:20.183Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:28:20.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e543113-afd2-46e1-8577-76304a152e47 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:20.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e543113-afd2-46e1-8577-76304a152e47 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44d9b9e5 2021-05-04T00:28:20.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e543113-afd2-46e1-8577-76304a152e47 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:20.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e543113-afd2-46e1-8577-76304a152e47 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ab498c5-ae2e-4753-b99e-72abe37d9a4c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ab498c5-ae2e-4753-b99e-72abe37d9a4c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3325d2fb 2021-05-04T00:28:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ab498c5-ae2e-4753-b99e-72abe37d9a4c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ab498c5-ae2e-4753-b99e-72abe37d9a4c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df382d53-ba71-42b6-831c-978f7421bf56 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df382d53-ba71-42b6-831c-978f7421bf56 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19e6d86d 2021-05-04T00:28:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df382d53-ba71-42b6-831c-978f7421bf56 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df382d53-ba71-42b6-831c-978f7421bf56 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:29.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b15b79e-7021-44d2-a217-bb9f4ba7df3a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:29.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b15b79e-7021-44d2-a217-bb9f4ba7df3a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f051aa9 2021-05-04T00:28:29.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b15b79e-7021-44d2-a217-bb9f4ba7df3a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:29.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b15b79e-7021-44d2-a217-bb9f4ba7df3a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:32.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=804c0d1a-e2fa-48e7-8fd0-d10eb00b3eb1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:32.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=804c0d1a-e2fa-48e7-8fd0-d10eb00b3eb1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@169a9e3d 2021-05-04T00:28:32.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=804c0d1a-e2fa-48e7-8fd0-d10eb00b3eb1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:32.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=804c0d1a-e2fa-48e7-8fd0-d10eb00b3eb1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:28:35.202Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:28:35.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c2af2bb-98fd-4dab-b86f-738120f7e32c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:35.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c2af2bb-98fd-4dab-b86f-738120f7e32c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bb5341f 2021-05-04T00:28:35.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c2af2bb-98fd-4dab-b86f-738120f7e32c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:35.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c2af2bb-98fd-4dab-b86f-738120f7e32c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:28:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:28:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:28:38.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=594d4baf-3cb6-472b-bb3e-8ae3cef81d89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:38.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=594d4baf-3cb6-472b-bb3e-8ae3cef81d89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24bfa0b4 2021-05-04T00:28:38.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=594d4baf-3cb6-472b-bb3e-8ae3cef81d89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:38.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=594d4baf-3cb6-472b-bb3e-8ae3cef81d89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:41.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5a6220f-b945-48e6-859b-1d5e978d50bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:41.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5a6220f-b945-48e6-859b-1d5e978d50bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@135826d8 2021-05-04T00:28:41.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5a6220f-b945-48e6-859b-1d5e978d50bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:41.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5a6220f-b945-48e6-859b-1d5e978d50bd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:44.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17149eff-092b-4a05-9db7-d4123c280f13 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:44.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17149eff-092b-4a05-9db7-d4123c280f13 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@517139f8 2021-05-04T00:28:44.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17149eff-092b-4a05-9db7-d4123c280f13 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:44.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17149eff-092b-4a05-9db7-d4123c280f13 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:47.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=502b0f7d-e757-40bd-9801-be77e27b5b92 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:47.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=502b0f7d-e757-40bd-9801-be77e27b5b92 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73c6ea05 2021-05-04T00:28:47.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=502b0f7d-e757-40bd-9801-be77e27b5b92 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:47.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=502b0f7d-e757-40bd-9801-be77e27b5b92 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:28:50.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:28:50.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c853d061-a924-43ef-8bc1-fa9153fb639e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:50.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c853d061-a924-43ef-8bc1-fa9153fb639e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2aa54ec9 2021-05-04T00:28:50.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c853d061-a924-43ef-8bc1-fa9153fb639e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:50.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c853d061-a924-43ef-8bc1-fa9153fb639e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:53.028Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:28:53.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973567e8-b8a3-4935-82bf-440131eac18f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:53.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973567e8-b8a3-4935-82bf-440131eac18f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7819c3a3 2021-05-04T00:28:53.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973567e8-b8a3-4935-82bf-440131eac18f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:53.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973567e8-b8a3-4935-82bf-440131eac18f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d1bdb91-9ace-48e2-a462-e88520006724 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d1bdb91-9ace-48e2-a462-e88520006724 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@207f9ba1 2021-05-04T00:28:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d1bdb91-9ace-48e2-a462-e88520006724 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d1bdb91-9ace-48e2-a462-e88520006724 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:28:59.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d0d574-5a89-44ae-b1a1-9ec698831324 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:28:59.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d0d574-5a89-44ae-b1a1-9ec698831324 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@642dcb69 2021-05-04T00:28:59.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d0d574-5a89-44ae-b1a1-9ec698831324 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:28:59.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d0d574-5a89-44ae-b1a1-9ec698831324 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f687059-cab8-4406-93ac-23ad049fc759 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f687059-cab8-4406-93ac-23ad049fc759 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@eb2eaaf 2021-05-04T00:29:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f687059-cab8-4406-93ac-23ad049fc759 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f687059-cab8-4406-93ac-23ad049fc759 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:29:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f10363f5-0266-4afd-a6bd-118097efa586 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:05.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f10363f5-0266-4afd-a6bd-118097efa586 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44e1e93 2021-05-04T00:29:05.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f10363f5-0266-4afd-a6bd-118097efa586 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:05.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f10363f5-0266-4afd-a6bd-118097efa586 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:06.756Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:29:08.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6327a8f-82c3-46e1-bcc2-4d53ff1b2fff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:08.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6327a8f-82c3-46e1-bcc2-4d53ff1b2fff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10b5fb23 2021-05-04T00:29:08.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6327a8f-82c3-46e1-bcc2-4d53ff1b2fff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:08.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6327a8f-82c3-46e1-bcc2-4d53ff1b2fff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:11.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=859bd6ba-6740-4075-b571-616b177c97f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:11.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=859bd6ba-6740-4075-b571-616b177c97f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24c3ca10 2021-05-04T00:29:11.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=859bd6ba-6740-4075-b571-616b177c97f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:11.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=859bd6ba-6740-4075-b571-616b177c97f6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:14.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf1fd0f9-68a4-47af-9254-fa8bc40e07a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:14.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf1fd0f9-68a4-47af-9254-fa8bc40e07a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3dbdbbb5 2021-05-04T00:29:14.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf1fd0f9-68a4-47af-9254-fa8bc40e07a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:14.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf1fd0f9-68a4-47af-9254-fa8bc40e07a2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:17.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b2ee1de-5e94-46a3-aaf1-9ddfbb161d95 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:17.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b2ee1de-5e94-46a3-aaf1-9ddfbb161d95 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49f39017 2021-05-04T00:29:17.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b2ee1de-5e94-46a3-aaf1-9ddfbb161d95 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:17.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b2ee1de-5e94-46a3-aaf1-9ddfbb161d95 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:29:20.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:29:20.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=184b536a-decf-42d7-af12-539c84d33245 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:20.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=184b536a-decf-42d7-af12-539c84d33245 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76a016e9 2021-05-04T00:29:20.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=184b536a-decf-42d7-af12-539c84d33245 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:20.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=184b536a-decf-42d7-af12-539c84d33245 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:23.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=207a2067-286f-4c27-9818-4f163e605afd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:23.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=207a2067-286f-4c27-9818-4f163e605afd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c09c167 2021-05-04T00:29:23.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=207a2067-286f-4c27-9818-4f163e605afd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:23.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=207a2067-286f-4c27-9818-4f163e605afd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:26.548Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3b93899-6e94-4a49-bbd3-105eccb6ebd2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:26.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3b93899-6e94-4a49-bbd3-105eccb6ebd2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c39bb40 2021-05-04T00:29:26.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3b93899-6e94-4a49-bbd3-105eccb6ebd2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:26.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3b93899-6e94-4a49-bbd3-105eccb6ebd2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:29.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db74748b-72e4-4038-a05f-8d011e6fe8e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:29.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db74748b-72e4-4038-a05f-8d011e6fe8e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@499fe3f8 2021-05-04T00:29:29.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db74748b-72e4-4038-a05f-8d011e6fe8e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:29.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db74748b-72e4-4038-a05f-8d011e6fe8e0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:32.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fd640d4-904c-4010-828e-f54b9a5b862f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:32.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fd640d4-904c-4010-828e-f54b9a5b862f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ffeb37c 2021-05-04T00:29:32.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fd640d4-904c-4010-828e-f54b9a5b862f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:32.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fd640d4-904c-4010-828e-f54b9a5b862f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:29:35.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:29:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=913229ed-6be5-47ee-a3c1-862304e48297 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=913229ed-6be5-47ee-a3c1-862304e48297 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3dfe5f1d 2021-05-04T00:29:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=913229ed-6be5-47ee-a3c1-862304e48297 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=913229ed-6be5-47ee-a3c1-862304e48297 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:29:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:29:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:29:38.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=050329e1-c80f-40b5-a9fc-7d1d938504d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:38.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=050329e1-c80f-40b5-a9fc-7d1d938504d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ccc1059 2021-05-04T00:29:38.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=050329e1-c80f-40b5-a9fc-7d1d938504d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:38.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=050329e1-c80f-40b5-a9fc-7d1d938504d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:41.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3873442-bc41-4787-9352-be68b12a2a7d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:41.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3873442-bc41-4787-9352-be68b12a2a7d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a7abe1b 2021-05-04T00:29:41.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3873442-bc41-4787-9352-be68b12a2a7d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:41.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3873442-bc41-4787-9352-be68b12a2a7d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:44.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24a296fc-a4cc-482f-8c59-41d3be6e776d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:44.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24a296fc-a4cc-482f-8c59-41d3be6e776d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1055837b 2021-05-04T00:29:44.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24a296fc-a4cc-482f-8c59-41d3be6e776d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:44.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24a296fc-a4cc-482f-8c59-41d3be6e776d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:47.349Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd995b42-51f2-4e6d-8403-265605ae5851 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:47.349Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd995b42-51f2-4e6d-8403-265605ae5851 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f1c30db 2021-05-04T00:29:47.349Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd995b42-51f2-4e6d-8403-265605ae5851 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:47.349Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd995b42-51f2-4e6d-8403-265605ae5851 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:29:50.185Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:29:50.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64c70a23-46fa-4075-b357-8c226bdbfe11 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:50.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64c70a23-46fa-4075-b357-8c226bdbfe11 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@535eadb9 2021-05-04T00:29:50.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64c70a23-46fa-4075-b357-8c226bdbfe11 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:50.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64c70a23-46fa-4075-b357-8c226bdbfe11 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:52.078Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:29:53.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d38960-0fb5-4746-a9f7-5101a39e2d90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:53.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d38960-0fb5-4746-a9f7-5101a39e2d90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c5b4baf 2021-05-04T00:29:53.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d38960-0fb5-4746-a9f7-5101a39e2d90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:53.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5d38960-0fb5-4746-a9f7-5101a39e2d90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:56.325Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c672693-7249-4a73-a68a-f24fb5d39fd6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:56.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c672693-7249-4a73-a68a-f24fb5d39fd6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60a94504 2021-05-04T00:29:56.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c672693-7249-4a73-a68a-f24fb5d39fd6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:56.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c672693-7249-4a73-a68a-f24fb5d39fd6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:29:59.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b487472-1965-4cb2-a7a8-2c26edd5ec1a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:29:59.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b487472-1965-4cb2-a7a8-2c26edd5ec1a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e8d3b22 2021-05-04T00:29:59.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b487472-1965-4cb2-a7a8-2c26edd5ec1a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:29:59.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b487472-1965-4cb2-a7a8-2c26edd5ec1a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:02.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4e190e7-e4fa-4de6-8b8c-2bb0ae3e9e92 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:02.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4e190e7-e4fa-4de6-8b8c-2bb0ae3e9e92 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7818762f 2021-05-04T00:30:02.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4e190e7-e4fa-4de6-8b8c-2bb0ae3e9e92 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:02.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4e190e7-e4fa-4de6-8b8c-2bb0ae3e9e92 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:30:05.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:30:05.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a59309c9-4ae5-4c5f-bdc1-ea37bcfc532d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:05.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a59309c9-4ae5-4c5f-bdc1-ea37bcfc532d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e184cb0 2021-05-04T00:30:05.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a59309c9-4ae5-4c5f-bdc1-ea37bcfc532d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:05.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a59309c9-4ae5-4c5f-bdc1-ea37bcfc532d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:08.063Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node06dl2c2dikvdd23jo7de94oa8238 2021-05-04T00:30:08.064Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01e7rgbr0erq0alcipydyiddm243 2021-05-04T00:30:08.064Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0appr3ryezagx17yvkq9rmq6va241 2021-05-04T00:30:08.064Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0mo9zq37r8kyyg1wl2m4eqf0d235 2021-05-04T00:30:08.064Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node02aj7p1itjagmtlgjj1igx6sr240 2021-05-04T00:30:08.064Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0134q72keas8so3y84a8zmg7o7239 2021-05-04T00:30:08.064Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01gpt7j4151eyf1bmtupip3tyi7234 2021-05-04T00:30:08.064Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01xv07ldctwaxn1dv3gpkzatvep236 2021-05-04T00:30:08.064Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node015i76sffh8gqs1l8lhgy7meuib242 2021-05-04T00:30:08.064Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01q5ee5l09b8jr212s4ceeq94b237 2021-05-04T00:30:08.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f9cf6c5-2f86-4cac-a3a6-741403a9e5c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:08.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f9cf6c5-2f86-4cac-a3a6-741403a9e5c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54b3a712 2021-05-04T00:30:08.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f9cf6c5-2f86-4cac-a3a6-741403a9e5c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:08.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f9cf6c5-2f86-4cac-a3a6-741403a9e5c6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:11.396Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed5ee470-d7c7-4014-8409-3d4f6e749360 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:11.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed5ee470-d7c7-4014-8409-3d4f6e749360 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6350d28d 2021-05-04T00:30:11.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed5ee470-d7c7-4014-8409-3d4f6e749360 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:11.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed5ee470-d7c7-4014-8409-3d4f6e749360 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:14.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=985964f1-6f69-455b-8fa4-ddc098e1e62e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:14.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=985964f1-6f69-455b-8fa4-ddc098e1e62e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@584269c6 2021-05-04T00:30:14.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=985964f1-6f69-455b-8fa4-ddc098e1e62e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:14.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=985964f1-6f69-455b-8fa4-ddc098e1e62e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:17.325Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7d7bf32-66a3-4be8-ac14-78025f9659d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:17.325Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7d7bf32-66a3-4be8-ac14-78025f9659d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6483d3b2 2021-05-04T00:30:17.325Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7d7bf32-66a3-4be8-ac14-78025f9659d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:17.325Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7d7bf32-66a3-4be8-ac14-78025f9659d7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:20.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:30:20.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:30:20.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40d72bdb-559f-4356-9ba7-15b85335d7a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:20.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40d72bdb-559f-4356-9ba7-15b85335d7a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2dc17307 2021-05-04T00:30:20.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40d72bdb-559f-4356-9ba7-15b85335d7a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:20.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40d72bdb-559f-4356-9ba7-15b85335d7a3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:23.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df81dfee-2c95-4cd7-9a53-fc3f23c6f4ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:23.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df81dfee-2c95-4cd7-9a53-fc3f23c6f4ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48600888 2021-05-04T00:30:23.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df81dfee-2c95-4cd7-9a53-fc3f23c6f4ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:23.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df81dfee-2c95-4cd7-9a53-fc3f23c6f4ba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:26.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deda60db-49d4-4bca-8e88-6e682f95ce5b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:26.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deda60db-49d4-4bca-8e88-6e682f95ce5b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@327541b 2021-05-04T00:30:26.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deda60db-49d4-4bca-8e88-6e682f95ce5b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:26.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deda60db-49d4-4bca-8e88-6e682f95ce5b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:29.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e45901f0-9bef-4a5d-8783-bb10d89822db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:29.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e45901f0-9bef-4a5d-8783-bb10d89822db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79c3098e 2021-05-04T00:30:29.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e45901f0-9bef-4a5d-8783-bb10d89822db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:29.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e45901f0-9bef-4a5d-8783-bb10d89822db PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:32.244Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebd37a4a-02a0-4818-9c46-00e921fb640a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:32.245Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebd37a4a-02a0-4818-9c46-00e921fb640a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5fbc04fa 2021-05-04T00:30:32.245Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebd37a4a-02a0-4818-9c46-00e921fb640a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:32.245Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebd37a4a-02a0-4818-9c46-00e921fb640a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:35.020Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfbd7bab-42ac-43c0-b2a7-5cb6471b4f02 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:35.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfbd7bab-42ac-43c0-b2a7-5cb6471b4f02 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17d073cd 2021-05-04T00:30:35.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfbd7bab-42ac-43c0-b2a7-5cb6471b4f02 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:35.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfbd7bab-42ac-43c0-b2a7-5cb6471b4f02 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:30:35.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:30:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:30:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:30:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:30:38.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb011a34-f10b-4079-95a3-da9410040003 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:38.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb011a34-f10b-4079-95a3-da9410040003 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e6e6aa0 2021-05-04T00:30:38.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb011a34-f10b-4079-95a3-da9410040003 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:38.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb011a34-f10b-4079-95a3-da9410040003 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:41.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b76e9ee-db8f-4388-aa9d-28bb70dbd21e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:41.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b76e9ee-db8f-4388-aa9d-28bb70dbd21e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f00899a 2021-05-04T00:30:41.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b76e9ee-db8f-4388-aa9d-28bb70dbd21e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:41.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b76e9ee-db8f-4388-aa9d-28bb70dbd21e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=096fcdb5-671f-4f16-80b3-0b4a72023a9f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=096fcdb5-671f-4f16-80b3-0b4a72023a9f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70a4cf54 2021-05-04T00:30:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=096fcdb5-671f-4f16-80b3-0b4a72023a9f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=096fcdb5-671f-4f16-80b3-0b4a72023a9f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:46.984Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=428ed731-6374-4589-ab73-90bf23d5831b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:46.984Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=428ed731-6374-4589-ab73-90bf23d5831b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64fe9496 2021-05-04T00:30:46.985Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=428ed731-6374-4589-ab73-90bf23d5831b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:46.985Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=428ed731-6374-4589-ab73-90bf23d5831b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:30:50.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:30:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa692015-c511-4575-9dbb-4f9e6885e667 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa692015-c511-4575-9dbb-4f9e6885e667 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3918c9c3 2021-05-04T00:30:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa692015-c511-4575-9dbb-4f9e6885e667 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa692015-c511-4575-9dbb-4f9e6885e667 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:52.051Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:30:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc2a0f1c-0b34-4afa-92c6-f8ee73c62299 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc2a0f1c-0b34-4afa-92c6-f8ee73c62299 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@721fcb23 2021-05-04T00:30:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc2a0f1c-0b34-4afa-92c6-f8ee73c62299 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc2a0f1c-0b34-4afa-92c6-f8ee73c62299 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:56.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50565dd0-070b-40d7-aaec-a0b9e6ea6689 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:56.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50565dd0-070b-40d7-aaec-a0b9e6ea6689 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2497645b 2021-05-04T00:30:56.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50565dd0-070b-40d7-aaec-a0b9e6ea6689 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:56.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50565dd0-070b-40d7-aaec-a0b9e6ea6689 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:30:59.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92075486-484d-4149-80a6-3cf6f32c49f3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:30:59.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92075486-484d-4149-80a6-3cf6f32c49f3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@345d0e7b 2021-05-04T00:30:59.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92075486-484d-4149-80a6-3cf6f32c49f3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:30:59.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92075486-484d-4149-80a6-3cf6f32c49f3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:02.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86f1e227-5d15-4dde-a00b-210da7ee4282 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:02.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86f1e227-5d15-4dde-a00b-210da7ee4282 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b748fe2 2021-05-04T00:31:02.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86f1e227-5d15-4dde-a00b-210da7ee4282 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:02.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86f1e227-5d15-4dde-a00b-210da7ee4282 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:31:05.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:31:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=63b1cb07-c599-4567-a76d-9b6907fb3601 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=63b1cb07-c599-4567-a76d-9b6907fb3601 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@473e137c 2021-05-04T00:31:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=63b1cb07-c599-4567-a76d-9b6907fb3601 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=63b1cb07-c599-4567-a76d-9b6907fb3601 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc27d203-43c2-4b7a-aaf7-2014ef699d9e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc27d203-43c2-4b7a-aaf7-2014ef699d9e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5894127c 2021-05-04T00:31:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc27d203-43c2-4b7a-aaf7-2014ef699d9e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc27d203-43c2-4b7a-aaf7-2014ef699d9e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebf3faff-c3c4-4d48-9fcb-45f02f4f4003 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebf3faff-c3c4-4d48-9fcb-45f02f4f4003 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@466e1c3 2021-05-04T00:31:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebf3faff-c3c4-4d48-9fcb-45f02f4f4003 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebf3faff-c3c4-4d48-9fcb-45f02f4f4003 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:14.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9f6437-c42a-494d-8aa7-7226d23acd44 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:14.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9f6437-c42a-494d-8aa7-7226d23acd44 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b826532 2021-05-04T00:31:14.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9f6437-c42a-494d-8aa7-7226d23acd44 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:14.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9f6437-c42a-494d-8aa7-7226d23acd44 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:17.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9deb6c7-5eb5-4b5e-abc3-348815e8fb31 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:17.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9deb6c7-5eb5-4b5e-abc3-348815e8fb31 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c903ecd 2021-05-04T00:31:17.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9deb6c7-5eb5-4b5e-abc3-348815e8fb31 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:17.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9deb6c7-5eb5-4b5e-abc3-348815e8fb31 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:31:20.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:31:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=384e7c86-5688-488c-b3e3-0086cbd174ee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=384e7c86-5688-488c-b3e3-0086cbd174ee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@246fd0d1 2021-05-04T00:31:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=384e7c86-5688-488c-b3e3-0086cbd174ee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=384e7c86-5688-488c-b3e3-0086cbd174ee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:23.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c53ac554-c8e3-474e-a4df-fd6da87fc2fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:23.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c53ac554-c8e3-474e-a4df-fd6da87fc2fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17b44566 2021-05-04T00:31:23.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c53ac554-c8e3-474e-a4df-fd6da87fc2fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:23.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c53ac554-c8e3-474e-a4df-fd6da87fc2fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:26.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c497e42-03cf-4eea-9a6c-166f2563ac42 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:26.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c497e42-03cf-4eea-9a6c-166f2563ac42 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f670957 2021-05-04T00:31:26.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c497e42-03cf-4eea-9a6c-166f2563ac42 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:26.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c497e42-03cf-4eea-9a6c-166f2563ac42 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:29.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8f2075c-479a-40b3-86c5-e40d7eb8f446 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:29.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8f2075c-479a-40b3-86c5-e40d7eb8f446 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@325c2420 2021-05-04T00:31:29.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8f2075c-479a-40b3-86c5-e40d7eb8f446 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:29.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8f2075c-479a-40b3-86c5-e40d7eb8f446 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:32.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7944dbcc-be8a-433d-a426-fb835c18f3e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:32.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7944dbcc-be8a-433d-a426-fb835c18f3e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68ef83e2 2021-05-04T00:31:32.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7944dbcc-be8a-433d-a426-fb835c18f3e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:32.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7944dbcc-be8a-433d-a426-fb835c18f3e0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:31:35.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:31:35.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a1fc378-2989-4292-a39f-5da090c1d929 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:35.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a1fc378-2989-4292-a39f-5da090c1d929 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d4574ec 2021-05-04T00:31:35.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a1fc378-2989-4292-a39f-5da090c1d929 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:35.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a1fc378-2989-4292-a39f-5da090c1d929 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:31:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:31:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:31:38.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a57fd1ed-bf15-4bc4-81c1-68150709b09a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:38.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a57fd1ed-bf15-4bc4-81c1-68150709b09a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@abff3a2 2021-05-04T00:31:38.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a57fd1ed-bf15-4bc4-81c1-68150709b09a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:38.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a57fd1ed-bf15-4bc4-81c1-68150709b09a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:41.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=298ff948-5d40-45dc-944c-9b2728f569a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:41.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=298ff948-5d40-45dc-944c-9b2728f569a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5991abf1 2021-05-04T00:31:41.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=298ff948-5d40-45dc-944c-9b2728f569a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:41.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=298ff948-5d40-45dc-944c-9b2728f569a6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaa6ad2e-c540-44c2-8e4d-8e2b0b043a89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaa6ad2e-c540-44c2-8e4d-8e2b0b043a89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4df5c3b2 2021-05-04T00:31:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaa6ad2e-c540-44c2-8e4d-8e2b0b043a89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaa6ad2e-c540-44c2-8e4d-8e2b0b043a89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:47.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a654e12-8140-4dab-ae0d-6ceab8ec444f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:47.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a654e12-8140-4dab-ae0d-6ceab8ec444f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3be12cf4 2021-05-04T00:31:47.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a654e12-8140-4dab-ae0d-6ceab8ec444f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:47.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a654e12-8140-4dab-ae0d-6ceab8ec444f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:50.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f9c5f4-0230-4a5b-a86f-8c37f45c0b9b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:50.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f9c5f4-0230-4a5b-a86f-8c37f45c0b9b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e36eac9 2021-05-04T00:31:50.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f9c5f4-0230-4a5b-a86f-8c37f45c0b9b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:50.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f9c5f4-0230-4a5b-a86f-8c37f45c0b9b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:31:50.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:31:52.051Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:31:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b871ada-a07c-429d-9672-cce88b790dc5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b871ada-a07c-429d-9672-cce88b790dc5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a2c1edd 2021-05-04T00:31:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b871ada-a07c-429d-9672-cce88b790dc5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b871ada-a07c-429d-9672-cce88b790dc5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:56.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da39a00c-2392-4651-9c51-3b527cb58cd5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:56.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da39a00c-2392-4651-9c51-3b527cb58cd5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79a7a679 2021-05-04T00:31:56.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da39a00c-2392-4651-9c51-3b527cb58cd5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:56.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da39a00c-2392-4651-9c51-3b527cb58cd5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:31:59.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5212da51-4da6-40d1-b104-158a8132a8f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:31:59.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5212da51-4da6-40d1-b104-158a8132a8f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b13a335 2021-05-04T00:31:59.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5212da51-4da6-40d1-b104-158a8132a8f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:31:59.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5212da51-4da6-40d1-b104-158a8132a8f1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:02.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c891be53-95ee-4799-9f73-c6c407adba9d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:02.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c891be53-95ee-4799-9f73-c6c407adba9d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@320a683c 2021-05-04T00:32:02.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c891be53-95ee-4799-9f73-c6c407adba9d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:02.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c891be53-95ee-4799-9f73-c6c407adba9d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:32:05.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:32:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d19be1cb-fe96-4969-b278-992cdc9e6028 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d19be1cb-fe96-4969-b278-992cdc9e6028 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48aa4256 2021-05-04T00:32:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d19be1cb-fe96-4969-b278-992cdc9e6028 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d19be1cb-fe96-4969-b278-992cdc9e6028 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:08.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=608d7360-111a-4ad7-ab57-3821907f72db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:08.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=608d7360-111a-4ad7-ab57-3821907f72db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c7ab154 2021-05-04T00:32:08.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=608d7360-111a-4ad7-ab57-3821907f72db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:08.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=608d7360-111a-4ad7-ab57-3821907f72db PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:11.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e55e6d87-88ee-4ccf-a0aa-76c2c76b29f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:11.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e55e6d87-88ee-4ccf-a0aa-76c2c76b29f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5aa6b1e5 2021-05-04T00:32:11.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e55e6d87-88ee-4ccf-a0aa-76c2c76b29f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:11.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e55e6d87-88ee-4ccf-a0aa-76c2c76b29f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:14.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=148aaf21-43ec-4621-9656-99f08ccf39f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=148aaf21-43ec-4621-9656-99f08ccf39f8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b8c6905 2021-05-04T00:32:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=148aaf21-43ec-4621-9656-99f08ccf39f8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=148aaf21-43ec-4621-9656-99f08ccf39f8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:17.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9b9c13d-265c-4e92-a3ec-52880fc445e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:17.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9b9c13d-265c-4e92-a3ec-52880fc445e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@212e34d0 2021-05-04T00:32:17.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9b9c13d-265c-4e92-a3ec-52880fc445e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:17.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9b9c13d-265c-4e92-a3ec-52880fc445e6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:32:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:32:20.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a2e1798-4cfc-480e-815a-5252aba6272b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:20.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a2e1798-4cfc-480e-815a-5252aba6272b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7807c1b0 2021-05-04T00:32:20.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a2e1798-4cfc-480e-815a-5252aba6272b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:20.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a2e1798-4cfc-480e-815a-5252aba6272b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=560d77f4-4576-4158-a652-8f2712d02f12 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=560d77f4-4576-4158-a652-8f2712d02f12 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17c17ecc 2021-05-04T00:32:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=560d77f4-4576-4158-a652-8f2712d02f12 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=560d77f4-4576-4158-a652-8f2712d02f12 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:26.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcb06682-6cec-4c23-b9b7-6651cc4d3040 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:26.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcb06682-6cec-4c23-b9b7-6651cc4d3040 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4fdde970 2021-05-04T00:32:26.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcb06682-6cec-4c23-b9b7-6651cc4d3040 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:26.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcb06682-6cec-4c23-b9b7-6651cc4d3040 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d7ab8ee-f5b2-4913-adc2-12624ab2d336 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d7ab8ee-f5b2-4913-adc2-12624ab2d336 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f85dcef 2021-05-04T00:32:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d7ab8ee-f5b2-4913-adc2-12624ab2d336 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d7ab8ee-f5b2-4913-adc2-12624ab2d336 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:31.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87f14718-b53e-49e8-93d0-a7b9b6be9c32 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:31.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87f14718-b53e-49e8-93d0-a7b9b6be9c32 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59c984ee 2021-05-04T00:32:31.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87f14718-b53e-49e8-93d0-a7b9b6be9c32 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:31.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87f14718-b53e-49e8-93d0-a7b9b6be9c32 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:32:35.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:32:35.305Z [qtp580718781-20] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=f297a26f-6b53-492f-a50b-a1758444b18a InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:32:35.304559Z InvocationID=424bf267-7e5b-4486-932c-6275ed2e8bba LogTimestamp=2021-05-04T00:32:35.304716Z Entering 2021-05-04T00:32:35.305Z [qtp580718781-20] INFO o.o.sdc.be.filters.BeServletFilter RequestID=f297a26f-6b53-492f-a50b-a1758444b18a InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:32:35.304559Z InvocationID=424bf267-7e5b-4486-932c-6275ed2e8bba LogTimestamp=2021-05-04T00:32:35.304716Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:32:35.309Z [qtp580718781-20] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:32:35.305Z InvocationID=2d4db48f-88ef-4c20-bff2-ee8d35f6d087 LogTimestamp=2021-05-04T00:32:35.309492Z ResponseDesc=OK Exiting. 2021-05-04T00:32:35.309Z [qtp580718781-20] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:32:35.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a70baf25-634b-4174-9adc-f5b3d89bed4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:35.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a70baf25-634b-4174-9adc-f5b3d89bed4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4562b605 2021-05-04T00:32:35.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a70baf25-634b-4174-9adc-f5b3d89bed4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:35.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a70baf25-634b-4174-9adc-f5b3d89bed4e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:37.005Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=8c5c3456-1a96-46c1-84c0-d9435b32e952 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:32:37.005388Z InvocationID=54d52b6c-a65e-4eca-b3b0-5fd5ccef768c LogTimestamp=2021-05-04T00:32:37.005468Z Entering 2021-05-04T00:32:37.005Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=8c5c3456-1a96-46c1-84c0-d9435b32e952 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:32:37.005388Z InvocationID=54d52b6c-a65e-4eca-b3b0-5fd5ccef768c LogTimestamp=2021-05-04T00:32:37.005468Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:32:37.012Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:32:37.006Z InvocationID=47554ba2-533e-478c-b2af-aeabc133a2f8 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/kQWYXIs4Vkv8eYWx will send credentials over a clear channel. 2021-05-04T00:32:37.012Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:32:37.006Z InvocationID=47554ba2-533e-478c-b2af-aeabc133a2f8 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/kQWYXIs4Vkv8eYWx (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:32:37.038Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:32:37.006Z InvocationID=47554ba2-533e-478c-b2af-aeabc133a2f8 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:32:37.102Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "kQWYXIs4Vkv8eYWx" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:32:37.006Z InvocationID=47554ba2-533e-478c-b2af-aeabc133a2f8 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/kQWYXIs4Vkv8eYWx will send credentials over a clear channel. 2021-05-04T00:32:37.102Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "kQWYXIs4Vkv8eYWx" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:32:37.006Z InvocationID=47554ba2-533e-478c-b2af-aeabc133a2f8 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/kQWYXIs4Vkv8eYWx (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:32:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:32:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:32:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:32:37.117Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "kQWYXIs4Vkv8eYWx" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:32:37.006Z InvocationID=47554ba2-533e-478c-b2af-aeabc133a2f8 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:32:37.161Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "cds" API_KEY = "kQWYXIs4Vkv8eYWx" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=155 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:32:37.006Z InvocationID=47554ba2-533e-478c-b2af-aeabc133a2f8 LogTimestamp=2021-05-04T00:32:37.161383Z ResponseDesc=OK Exiting. 2021-05-04T00:32:37.161Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:32:38.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a17a5143-341d-453b-b1d7-8297da6fbec0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:38.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a17a5143-341d-453b-b1d7-8297da6fbec0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@655441b0 2021-05-04T00:32:38.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a17a5143-341d-453b-b1d7-8297da6fbec0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:38.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a17a5143-341d-453b-b1d7-8297da6fbec0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:41.253Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28c0ada7-f866-46d0-95b1-e55d9ff06b23 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:41.253Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28c0ada7-f866-46d0-95b1-e55d9ff06b23 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54879b51 2021-05-04T00:32:41.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28c0ada7-f866-46d0-95b1-e55d9ff06b23 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:41.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28c0ada7-f866-46d0-95b1-e55d9ff06b23 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:44.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec837a4b-560f-47e5-a161-9b0fa38f9d79 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:44.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec837a4b-560f-47e5-a161-9b0fa38f9d79 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41fc8bdb 2021-05-04T00:32:44.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec837a4b-560f-47e5-a161-9b0fa38f9d79 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:44.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec837a4b-560f-47e5-a161-9b0fa38f9d79 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:46.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17456bd2-4a10-4b2c-b5f4-7b27d603be5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:46.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17456bd2-4a10-4b2c-b5f4-7b27d603be5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@117edf7f 2021-05-04T00:32:46.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17456bd2-4a10-4b2c-b5f4-7b27d603be5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:46.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17456bd2-4a10-4b2c-b5f4-7b27d603be5f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:50.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbd250fa-e9d2-4f4e-b997-45ea60315723 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:50.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbd250fa-e9d2-4f4e-b997-45ea60315723 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56605521 2021-05-04T00:32:50.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbd250fa-e9d2-4f4e-b997-45ea60315723 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:50.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbd250fa-e9d2-4f4e-b997-45ea60315723 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:32:50.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:32:52.050Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:32:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38eebb13-3c8b-43bb-ba35-89df343dc616 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38eebb13-3c8b-43bb-ba35-89df343dc616 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e5501af 2021-05-04T00:32:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38eebb13-3c8b-43bb-ba35-89df343dc616 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38eebb13-3c8b-43bb-ba35-89df343dc616 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:56.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c425c28-f645-49b4-85ed-b969a33f79fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:56.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c425c28-f645-49b4-85ed-b969a33f79fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bd2e7ed 2021-05-04T00:32:56.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c425c28-f645-49b4-85ed-b969a33f79fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:56.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c425c28-f645-49b4-85ed-b969a33f79fa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:32:59.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3acf14a-8823-44f0-b928-45119890c6e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:32:59.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3acf14a-8823-44f0-b928-45119890c6e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c3056cf 2021-05-04T00:32:59.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3acf14a-8823-44f0-b928-45119890c6e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:32:59.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3acf14a-8823-44f0-b928-45119890c6e7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:02.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0a4a14e-1db6-439c-adf3-c0d7f4b0edd0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:02.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0a4a14e-1db6-439c-adf3-c0d7f4b0edd0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@282ddd35 2021-05-04T00:33:02.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0a4a14e-1db6-439c-adf3-c0d7f4b0edd0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:02.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0a4a14e-1db6-439c-adf3-c0d7f4b0edd0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:33:05.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:33:05.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0526ae18-7ede-45bc-b33e-b8b40f13cd2f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:05.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0526ae18-7ede-45bc-b33e-b8b40f13cd2f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e63f9e6 2021-05-04T00:33:05.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0526ae18-7ede-45bc-b33e-b8b40f13cd2f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:05.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0526ae18-7ede-45bc-b33e-b8b40f13cd2f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aedf14e0-888f-4718-bdce-be952513805c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aedf14e0-888f-4718-bdce-be952513805c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32829c48 2021-05-04T00:33:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aedf14e0-888f-4718-bdce-be952513805c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aedf14e0-888f-4718-bdce-be952513805c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:11.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2a92b69-9020-4815-8226-5aee5d1e9033 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:11.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2a92b69-9020-4815-8226-5aee5d1e9033 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c7a32f9 2021-05-04T00:33:11.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2a92b69-9020-4815-8226-5aee5d1e9033 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:11.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2a92b69-9020-4815-8226-5aee5d1e9033 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:14.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39c3588c-01c9-4de1-80c9-626d3d06c5f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:14.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39c3588c-01c9-4de1-80c9-626d3d06c5f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@af88a36 2021-05-04T00:33:14.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39c3588c-01c9-4de1-80c9-626d3d06c5f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:14.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39c3588c-01c9-4de1-80c9-626d3d06c5f1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:17.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f5f7dce-59dc-4024-a2b3-e72993c49f70 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:17.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f5f7dce-59dc-4024-a2b3-e72993c49f70 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b75ee46 2021-05-04T00:33:17.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f5f7dce-59dc-4024-a2b3-e72993c49f70 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:17.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f5f7dce-59dc-4024-a2b3-e72993c49f70 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:33:20.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:33:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89487430-3603-4b80-a443-efcb0eb3d591 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89487430-3603-4b80-a443-efcb0eb3d591 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d3357bc 2021-05-04T00:33:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89487430-3603-4b80-a443-efcb0eb3d591 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89487430-3603-4b80-a443-efcb0eb3d591 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:23.332Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7b07bf7-b005-439f-afd0-605ae3d51d97 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:23.332Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7b07bf7-b005-439f-afd0-605ae3d51d97 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10434119 2021-05-04T00:33:23.332Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7b07bf7-b005-439f-afd0-605ae3d51d97 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:23.332Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7b07bf7-b005-439f-afd0-605ae3d51d97 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:25.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6fc6433-593b-4a44-91a3-062d915a300c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:25.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6fc6433-593b-4a44-91a3-062d915a300c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e24c01d 2021-05-04T00:33:25.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6fc6433-593b-4a44-91a3-062d915a300c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:25.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6fc6433-593b-4a44-91a3-062d915a300c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:29.980Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a708ede6-6a9f-4c21-9174-e913d280ae76 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:29.981Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a708ede6-6a9f-4c21-9174-e913d280ae76 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5aa71241 2021-05-04T00:33:29.981Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a708ede6-6a9f-4c21-9174-e913d280ae76 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:29.981Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a708ede6-6a9f-4c21-9174-e913d280ae76 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:32.984Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a708ede6-6a9f-4c21-9174-e913d280ae76 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:33:32.985Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a708ede6-6a9f-4c21-9174-e913d280ae76 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:33:32.986Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a708ede6-6a9f-4c21-9174-e913d280ae76 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:33:32.986Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a708ede6-6a9f-4c21-9174-e913d280ae76 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:33:33.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:33:33.205Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:33:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:33:35.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:33:36.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:33:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:33:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:33:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:33:38.001Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:33:38.003Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T00:33:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:33:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:33:44.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee90cede-dec5-417f-928a-683e9c823d1e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:44.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee90cede-dec5-417f-928a-683e9c823d1e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4dbe301e 2021-05-04T00:33:44.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee90cede-dec5-417f-928a-683e9c823d1e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:44.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee90cede-dec5-417f-928a-683e9c823d1e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:44.306Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:33:44.504Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ee90cede-dec5-417f-928a-683e9c823d1e ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:33:44.505Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ee90cede-dec5-417f-928a-683e9c823d1e ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:33:45.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a95a4444-bde2-4cb3-920d-ab9e4e4c3a76 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:45.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a95a4444-bde2-4cb3-920d-ab9e4e4c3a76 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f23efe3 2021-05-04T00:33:45.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a95a4444-bde2-4cb3-920d-ab9e4e4c3a76 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:45.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a95a4444-bde2-4cb3-920d-ab9e4e4c3a76 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:46.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=854b35b8-ebc4-4185-9158-abf646628398 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:46.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=854b35b8-ebc4-4185-9158-abf646628398 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5df4d75c 2021-05-04T00:33:46.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=854b35b8-ebc4-4185-9158-abf646628398 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:46.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=854b35b8-ebc4-4185-9158-abf646628398 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:47.691Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c685614d-b982-4520-8a4e-9f7e1e28c886 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:47.691Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c685614d-b982-4520-8a4e-9f7e1e28c886 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6176223c 2021-05-04T00:33:47.691Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c685614d-b982-4520-8a4e-9f7e1e28c886 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:47.691Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c685614d-b982-4520-8a4e-9f7e1e28c886 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:49.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24a98cdb-f7b9-4031-b112-2db87c55d009 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:49.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24a98cdb-f7b9-4031-b112-2db87c55d009 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59e8ab18 2021-05-04T00:33:49.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24a98cdb-f7b9-4031-b112-2db87c55d009 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:49.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24a98cdb-f7b9-4031-b112-2db87c55d009 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:33:50.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:33:50.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8c448f-439b-490a-a98f-cede58ee76cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:50.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8c448f-439b-490a-a98f-cede58ee76cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79141b31 2021-05-04T00:33:50.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8c448f-439b-490a-a98f-cede58ee76cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:50.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8c448f-439b-490a-a98f-cede58ee76cf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:51.997Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7186e9ca-a88e-46d0-8305-c535eaac038e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:51.997Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7186e9ca-a88e-46d0-8305-c535eaac038e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a16d78 2021-05-04T00:33:51.997Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7186e9ca-a88e-46d0-8305-c535eaac038e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:51.997Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7186e9ca-a88e-46d0-8305-c535eaac038e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:52.039Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:33:53.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da218b04-25f1-410a-ab0a-8fd74fbb4cc9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:53.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da218b04-25f1-410a-ab0a-8fd74fbb4cc9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5296d8d2 2021-05-04T00:33:53.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da218b04-25f1-410a-ab0a-8fd74fbb4cc9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:53.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da218b04-25f1-410a-ab0a-8fd74fbb4cc9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:56.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e30f65c-e393-4c15-8e82-93466cc9a3fc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:56.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e30f65c-e393-4c15-8e82-93466cc9a3fc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4eba19f3 2021-05-04T00:33:56.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e30f65c-e393-4c15-8e82-93466cc9a3fc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:56.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e30f65c-e393-4c15-8e82-93466cc9a3fc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:33:59.024Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aa213c2-7862-4972-971c-7e68b1c242d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:33:59.024Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aa213c2-7862-4972-971c-7e68b1c242d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17ae4e86 2021-05-04T00:33:59.024Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aa213c2-7862-4972-971c-7e68b1c242d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:33:59.024Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aa213c2-7862-4972-971c-7e68b1c242d6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:02.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc038ba3-225b-4b33-a235-2a0360c001f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:02.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc038ba3-225b-4b33-a235-2a0360c001f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1eed3ba6 2021-05-04T00:34:02.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc038ba3-225b-4b33-a235-2a0360c001f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:02.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc038ba3-225b-4b33-a235-2a0360c001f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:34:05.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9806de79-fc63-4cf8-a5c9-d0bb61599a70 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:05.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9806de79-fc63-4cf8-a5c9-d0bb61599a70 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ad920e1 2021-05-04T00:34:05.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9806de79-fc63-4cf8-a5c9-d0bb61599a70 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:05.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9806de79-fc63-4cf8-a5c9-d0bb61599a70 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:08.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50b53354-9735-4a29-9f05-83367c0e0715 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:08.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50b53354-9735-4a29-9f05-83367c0e0715 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6432dbc8 2021-05-04T00:34:08.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50b53354-9735-4a29-9f05-83367c0e0715 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:08.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50b53354-9735-4a29-9f05-83367c0e0715 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:11.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3c0da37-6824-42ab-b912-757a72ef762a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:11.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3c0da37-6824-42ab-b912-757a72ef762a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10d63b6 2021-05-04T00:34:11.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3c0da37-6824-42ab-b912-757a72ef762a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:11.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3c0da37-6824-42ab-b912-757a72ef762a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:13.169Z [UEB-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend probably lost connectivity to UEB Cluster. Please check the logs for more information. 2021-05-04T00:34:14.564Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c3c0da37-6824-42ab-b912-757a72ef762a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:34:14.564Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c3c0da37-6824-42ab-b912-757a72ef762a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:34:14.565Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c3c0da37-6824-42ab-b912-757a72ef762a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:34:14.565Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c3c0da37-6824-42ab-b912-757a72ef762a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:34:16.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50bde67b-8fad-4805-b247-8c33dedd433b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:16.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50bde67b-8fad-4805-b247-8c33dedd433b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@764a583a 2021-05-04T00:34:16.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50bde67b-8fad-4805-b247-8c33dedd433b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:16.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50bde67b-8fad-4805-b247-8c33dedd433b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:16.821Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=50bde67b-8fad-4805-b247-8c33dedd433b ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:34:16.821Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=50bde67b-8fad-4805-b247-8c33dedd433b ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:34:17.622Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e09b1a4-4beb-4729-95d3-ac66921184d0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:17.622Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e09b1a4-4beb-4729-95d3-ac66921184d0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37b9cfcd 2021-05-04T00:34:17.622Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e09b1a4-4beb-4729-95d3-ac66921184d0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:17.622Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e09b1a4-4beb-4729-95d3-ac66921184d0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:20.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b746341-4919-4e06-90a9-68f6aadaac7d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:20.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b746341-4919-4e06-90a9-68f6aadaac7d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5dc31a5f 2021-05-04T00:34:20.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b746341-4919-4e06-90a9-68f6aadaac7d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:20.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b746341-4919-4e06-90a9-68f6aadaac7d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:23.585Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=197de293-fb5d-47d6-b3bd-514c638357ad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:23.585Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=197de293-fb5d-47d6-b3bd-514c638357ad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@796c0255 2021-05-04T00:34:23.585Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=197de293-fb5d-47d6-b3bd-514c638357ad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:23.585Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=197de293-fb5d-47d6-b3bd-514c638357ad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:26.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8924e78f-0ad9-42d4-aac4-ec570d0caef4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:26.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8924e78f-0ad9-42d4-aac4-ec570d0caef4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b24b32 2021-05-04T00:34:26.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8924e78f-0ad9-42d4-aac4-ec570d0caef4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:26.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8924e78f-0ad9-42d4-aac4-ec570d0caef4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:29.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d824fc43-de2d-45a0-b34f-a67fdf4db4e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:29.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d824fc43-de2d-45a0-b34f-a67fdf4db4e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@780c2b6e 2021-05-04T00:34:29.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d824fc43-de2d-45a0-b34f-a67fdf4db4e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:29.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d824fc43-de2d-45a0-b34f-a67fdf4db4e0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:29.447Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:34:29.452Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:34:32.950Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f51a44da-0c64-46c1-8b43-73a3eb89d24f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:32.950Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f51a44da-0c64-46c1-8b43-73a3eb89d24f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3dd8d7d2 2021-05-04T00:34:32.950Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f51a44da-0c64-46c1-8b43-73a3eb89d24f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:32.950Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f51a44da-0c64-46c1-8b43-73a3eb89d24f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:35.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0712b2f-c195-420e-b818-b3875846fd2b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:35.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0712b2f-c195-420e-b818-b3875846fd2b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@243f1496 2021-05-04T00:34:35.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0712b2f-c195-420e-b818-b3875846fd2b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:35.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0712b2f-c195-420e-b818-b3875846fd2b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:34:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:34:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:34:38.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f35d1618-8c9e-4036-bf20-53596f56a3bf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:38.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f35d1618-8c9e-4036-bf20-53596f56a3bf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@374720f8 2021-05-04T00:34:38.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f35d1618-8c9e-4036-bf20-53596f56a3bf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:38.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f35d1618-8c9e-4036-bf20-53596f56a3bf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:41.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03d199a-ecf4-4182-a8a2-887278598a47 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:41.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03d199a-ecf4-4182-a8a2-887278598a47 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e093a25 2021-05-04T00:34:41.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03d199a-ecf4-4182-a8a2-887278598a47 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:41.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03d199a-ecf4-4182-a8a2-887278598a47 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:46.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86907902-b2c4-4ba1-a1b9-b5ca1bcd8c8b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:46.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86907902-b2c4-4ba1-a1b9-b5ca1bcd8c8b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55140e71 2021-05-04T00:34:46.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86907902-b2c4-4ba1-a1b9-b5ca1bcd8c8b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:46.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86907902-b2c4-4ba1-a1b9-b5ca1bcd8c8b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:47.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a85af6e-b731-46a5-ad10-249731e8b328 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:47.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a85af6e-b731-46a5-ad10-249731e8b328 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9532be1 2021-05-04T00:34:47.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a85af6e-b731-46a5-ad10-249731e8b328 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:47.255Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a85af6e-b731-46a5-ad10-249731e8b328 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=226e5560-7009-449a-9c8d-d43316aff723 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=226e5560-7009-449a-9c8d-d43316aff723 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79d8ad3 2021-05-04T00:34:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=226e5560-7009-449a-9c8d-d43316aff723 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=226e5560-7009-449a-9c8d-d43316aff723 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cb3260-b99c-441e-ae92-73a99af76313 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cb3260-b99c-441e-ae92-73a99af76313 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60a25017 2021-05-04T00:34:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cb3260-b99c-441e-ae92-73a99af76313 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4cb3260-b99c-441e-ae92-73a99af76313 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:56.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb4e669e-8166-4d83-9666-86b128d6f157 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:56.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb4e669e-8166-4d83-9666-86b128d6f157 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15b0bdf4 2021-05-04T00:34:56.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb4e669e-8166-4d83-9666-86b128d6f157 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:56.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb4e669e-8166-4d83-9666-86b128d6f157 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:34:59.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27db884d-6c98-45a7-bba5-922e7fb8f1ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:34:59.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27db884d-6c98-45a7-bba5-922e7fb8f1ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52c9080 2021-05-04T00:34:59.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27db884d-6c98-45a7-bba5-922e7fb8f1ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:34:59.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27db884d-6c98-45a7-bba5-922e7fb8f1ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:02.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2532cb3-6b20-4fa2-9224-318619f9b1fc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:02.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2532cb3-6b20-4fa2-9224-318619f9b1fc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e8c59e4 2021-05-04T00:35:02.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2532cb3-6b20-4fa2-9224-318619f9b1fc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:02.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2532cb3-6b20-4fa2-9224-318619f9b1fc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:05.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78313e6f-b1da-4196-8556-a0c57882e331 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:05.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78313e6f-b1da-4196-8556-a0c57882e331 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1cdc8eca 2021-05-04T00:35:05.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78313e6f-b1da-4196-8556-a0c57882e331 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:05.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78313e6f-b1da-4196-8556-a0c57882e331 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:08.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b36cfc5e-7f84-45ca-9b31-342a25cb4add PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:08.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b36cfc5e-7f84-45ca-9b31-342a25cb4add PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c13dfb8 2021-05-04T00:35:08.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b36cfc5e-7f84-45ca-9b31-342a25cb4add PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:08.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b36cfc5e-7f84-45ca-9b31-342a25cb4add PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:11.698Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7cf2067-8131-4673-b1fc-f24a2395de4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:11.698Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7cf2067-8131-4673-b1fc-f24a2395de4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d5ae62a 2021-05-04T00:35:11.698Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7cf2067-8131-4673-b1fc-f24a2395de4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:11.698Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7cf2067-8131-4673-b1fc-f24a2395de4a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:14.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f739daf7-3209-4734-82a2-be211c777131 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:14.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f739daf7-3209-4734-82a2-be211c777131 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6195c11c 2021-05-04T00:35:14.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f739daf7-3209-4734-82a2-be211c777131 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:14.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f739daf7-3209-4734-82a2-be211c777131 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:17.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=645c8bf2-d4ef-4ada-8ffd-38aafe9ccc9b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:17.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=645c8bf2-d4ef-4ada-8ffd-38aafe9ccc9b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1947b93a 2021-05-04T00:35:17.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=645c8bf2-d4ef-4ada-8ffd-38aafe9ccc9b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:17.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=645c8bf2-d4ef-4ada-8ffd-38aafe9ccc9b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:20.888Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e64c4ad-6b39-4de3-9f45-c5558dff6d69 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:20.888Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e64c4ad-6b39-4de3-9f45-c5558dff6d69 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b3b4bb5 2021-05-04T00:35:20.888Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e64c4ad-6b39-4de3-9f45-c5558dff6d69 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:20.888Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e64c4ad-6b39-4de3-9f45-c5558dff6d69 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:23.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7330dd27-e58c-4563-a064-702e8feedfea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:23.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7330dd27-e58c-4563-a064-702e8feedfea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@316a874e 2021-05-04T00:35:23.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7330dd27-e58c-4563-a064-702e8feedfea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:23.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7330dd27-e58c-4563-a064-702e8feedfea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:26.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca7d09e9-6ca8-4489-b6de-ad843930b9d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:26.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca7d09e9-6ca8-4489-b6de-ad843930b9d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e382d5e 2021-05-04T00:35:26.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca7d09e9-6ca8-4489-b6de-ad843930b9d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:26.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca7d09e9-6ca8-4489-b6de-ad843930b9d7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:29.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bdab2f3-2d80-4f02-978f-d02502df1852 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bdab2f3-2d80-4f02-978f-d02502df1852 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a92c5ca 2021-05-04T00:35:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bdab2f3-2d80-4f02-978f-d02502df1852 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bdab2f3-2d80-4f02-978f-d02502df1852 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:31.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e819779-73a9-42d3-8db6-e062453e83b2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:31.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e819779-73a9-42d3-8db6-e062453e83b2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ba54bce 2021-05-04T00:35:31.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e819779-73a9-42d3-8db6-e062453e83b2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:31.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e819779-73a9-42d3-8db6-e062453e83b2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:35.919Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6588fa9e-9ba6-40d9-a0b0-2930b4cafb0b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:35.919Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6588fa9e-9ba6-40d9-a0b0-2930b4cafb0b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e10ed03 2021-05-04T00:35:35.919Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6588fa9e-9ba6-40d9-a0b0-2930b4cafb0b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:35.919Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6588fa9e-9ba6-40d9-a0b0-2930b4cafb0b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:38.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25f212e8-096a-49ef-a622-11611efb9b6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:38.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25f212e8-096a-49ef-a622-11611efb9b6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51fd7f42 2021-05-04T00:35:38.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25f212e8-096a-49ef-a622-11611efb9b6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:38.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25f212e8-096a-49ef-a622-11611efb9b6a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d9664bc-20ca-4263-94f2-8404b28ad5bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d9664bc-20ca-4263-94f2-8404b28ad5bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d8d5c22 2021-05-04T00:35:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d9664bc-20ca-4263-94f2-8404b28ad5bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d9664bc-20ca-4263-94f2-8404b28ad5bd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:44.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98318ece-bfe9-4d6a-905a-378418bfaaaa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:44.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98318ece-bfe9-4d6a-905a-378418bfaaaa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c578f66 2021-05-04T00:35:44.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98318ece-bfe9-4d6a-905a-378418bfaaaa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:44.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98318ece-bfe9-4d6a-905a-378418bfaaaa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:46.988Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6e32454-194c-4cb1-8313-728c95b7d4ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:46.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6e32454-194c-4cb1-8313-728c95b7d4ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5546151a 2021-05-04T00:35:46.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6e32454-194c-4cb1-8313-728c95b7d4ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:46.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6e32454-194c-4cb1-8313-728c95b7d4ba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:51.685Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76b87eb9-67da-42fe-b282-b215bb35c86e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:51.685Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76b87eb9-67da-42fe-b282-b215bb35c86e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ffb679f 2021-05-04T00:35:51.685Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76b87eb9-67da-42fe-b282-b215bb35c86e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:51.685Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76b87eb9-67da-42fe-b282-b215bb35c86e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:53.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c9f2caa-240e-460a-9898-86d3c47ad77f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:53.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c9f2caa-240e-460a-9898-86d3c47ad77f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18604a0e 2021-05-04T00:35:53.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c9f2caa-240e-460a-9898-86d3c47ad77f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:53.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c9f2caa-240e-460a-9898-86d3c47ad77f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a89b2638-76a8-4c5a-a71f-4e4fbe62e085 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a89b2638-76a8-4c5a-a71f-4e4fbe62e085 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@184ccadb 2021-05-04T00:35:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a89b2638-76a8-4c5a-a71f-4e4fbe62e085 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a89b2638-76a8-4c5a-a71f-4e4fbe62e085 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:35:59.680Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b2ec8f5-67d1-4316-80ff-fd73a9da6ec3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:35:59.680Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b2ec8f5-67d1-4316-80ff-fd73a9da6ec3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7d47f20d 2021-05-04T00:35:59.680Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b2ec8f5-67d1-4316-80ff-fd73a9da6ec3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:35:59.680Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b2ec8f5-67d1-4316-80ff-fd73a9da6ec3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:02.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bce46fa6-4519-433e-825a-5dbd0df39da1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:02.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bce46fa6-4519-433e-825a-5dbd0df39da1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4934a5a5 2021-05-04T00:36:02.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bce46fa6-4519-433e-825a-5dbd0df39da1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:02.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bce46fa6-4519-433e-825a-5dbd0df39da1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:05.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e475202d-834d-4895-a0a1-1bc8f4fa0754 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:05.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e475202d-834d-4895-a0a1-1bc8f4fa0754 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@539eb6b7 2021-05-04T00:36:05.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e475202d-834d-4895-a0a1-1bc8f4fa0754 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:05.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e475202d-834d-4895-a0a1-1bc8f4fa0754 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:08.140Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a5016e8-d134-475b-a209-39970dd265cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:08.141Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a5016e8-d134-475b-a209-39970dd265cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d7ab40d 2021-05-04T00:36:08.141Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a5016e8-d134-475b-a209-39970dd265cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:08.141Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a5016e8-d134-475b-a209-39970dd265cf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:11.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16c12e16-76fb-4b57-9d89-031c3d06b843 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:11.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16c12e16-76fb-4b57-9d89-031c3d06b843 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6fefec35 2021-05-04T00:36:11.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16c12e16-76fb-4b57-9d89-031c3d06b843 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:11.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16c12e16-76fb-4b57-9d89-031c3d06b843 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:14.699Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afff8f65-5e20-4063-8452-5c05333912ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:14.699Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afff8f65-5e20-4063-8452-5c05333912ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39a65e0 2021-05-04T00:36:14.699Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afff8f65-5e20-4063-8452-5c05333912ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:14.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afff8f65-5e20-4063-8452-5c05333912ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:17.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce978116-084d-4a6a-ad1d-caa8a0f5205b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:17.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce978116-084d-4a6a-ad1d-caa8a0f5205b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2517502a 2021-05-04T00:36:17.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce978116-084d-4a6a-ad1d-caa8a0f5205b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:17.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce978116-084d-4a6a-ad1d-caa8a0f5205b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45224ea5-5c2e-43af-bab7-56df428df539 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45224ea5-5c2e-43af-bab7-56df428df539 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5db55259 2021-05-04T00:36:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45224ea5-5c2e-43af-bab7-56df428df539 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45224ea5-5c2e-43af-bab7-56df428df539 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:23.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49d963ca-20e7-4f27-b98d-e6cdcb67b88f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:23.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49d963ca-20e7-4f27-b98d-e6cdcb67b88f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4dab939e 2021-05-04T00:36:23.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49d963ca-20e7-4f27-b98d-e6cdcb67b88f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:23.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49d963ca-20e7-4f27-b98d-e6cdcb67b88f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:26.046Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7314468f-78a4-43f2-a3fa-b9dbf3e0a22c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:26.046Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7314468f-78a4-43f2-a3fa-b9dbf3e0a22c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d4e5aae 2021-05-04T00:36:26.047Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7314468f-78a4-43f2-a3fa-b9dbf3e0a22c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:26.047Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7314468f-78a4-43f2-a3fa-b9dbf3e0a22c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db1fa8b7-59c0-4e45-9d58-cd5eec33a2e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db1fa8b7-59c0-4e45-9d58-cd5eec33a2e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@247df2d 2021-05-04T00:36:29.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db1fa8b7-59c0-4e45-9d58-cd5eec33a2e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:29.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db1fa8b7-59c0-4e45-9d58-cd5eec33a2e0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:30.444Z [qtp580718781-18] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=7583604a-8d86-4fe6-a0c8-3d0c7d57aef9 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:36:30.443320Z InvocationID=0766c813-4ed6-4250-923d-e7ba085fec76 LogTimestamp=2021-05-04T00:36:30.443475Z Entering 2021-05-04T00:36:30.444Z [qtp580718781-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=7583604a-8d86-4fe6-a0c8-3d0c7d57aef9 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:36:30.443320Z InvocationID=0766c813-4ed6-4250-923d-e7ba085fec76 LogTimestamp=2021-05-04T00:36:30.443475Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:36:30.448Z [qtp580718781-18] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:36:30.444Z InvocationID=c2bec8c2-17a9-4c00-8c72-ac87d6e995d4 LogTimestamp=2021-05-04T00:36:30.448839Z ResponseDesc=OK Exiting. 2021-05-04T00:36:30.449Z [qtp580718781-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:36:32.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03cc1352-23a8-4211-877d-79205b9a4adf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:32.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03cc1352-23a8-4211-877d-79205b9a4adf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@759762a6 2021-05-04T00:36:32.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03cc1352-23a8-4211-877d-79205b9a4adf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:32.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03cc1352-23a8-4211-877d-79205b9a4adf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:35.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4420f5c0-3efb-4c3c-aab4-3e93e9f4dbb9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:35.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4420f5c0-3efb-4c3c-aab4-3e93e9f4dbb9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37c78d19 2021-05-04T00:36:35.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4420f5c0-3efb-4c3c-aab4-3e93e9f4dbb9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:35.427Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4420f5c0-3efb-4c3c-aab4-3e93e9f4dbb9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:38.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5026ad66-28ad-410a-a0f8-6a42ae57a8b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:38.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5026ad66-28ad-410a-a0f8-6a42ae57a8b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@790b4b15 2021-05-04T00:36:38.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5026ad66-28ad-410a-a0f8-6a42ae57a8b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:38.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5026ad66-28ad-410a-a0f8-6a42ae57a8b8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baa9e066-d595-4dfb-9f47-43c33ba0f5cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baa9e066-d595-4dfb-9f47-43c33ba0f5cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@149042bd 2021-05-04T00:36:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baa9e066-d595-4dfb-9f47-43c33ba0f5cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baa9e066-d595-4dfb-9f47-43c33ba0f5cb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:44.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=148a3099-cbba-4cd8-8231-72dff0aa2a7b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:44.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=148a3099-cbba-4cd8-8231-72dff0aa2a7b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33dd6709 2021-05-04T00:36:44.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=148a3099-cbba-4cd8-8231-72dff0aa2a7b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:44.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=148a3099-cbba-4cd8-8231-72dff0aa2a7b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:47.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de47a996-b279-456b-854d-d520a0692f17 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:47.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de47a996-b279-456b-854d-d520a0692f17 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38f99b2e 2021-05-04T00:36:47.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de47a996-b279-456b-854d-d520a0692f17 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:47.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de47a996-b279-456b-854d-d520a0692f17 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:50.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f88f81c-2a83-4190-88a7-2a7f187391d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:50.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f88f81c-2a83-4190-88a7-2a7f187391d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b092cc7 2021-05-04T00:36:50.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f88f81c-2a83-4190-88a7-2a7f187391d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:50.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f88f81c-2a83-4190-88a7-2a7f187391d6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:52.276Z [qtp580718781-17] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=89be80fe-f40c-48b3-a440-f85197bc939d InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:36:52.275903Z InvocationID=1627ec82-554b-44d6-bc83-b9cd0c79369d LogTimestamp=2021-05-04T00:36:52.275969Z Entering 2021-05-04T00:36:52.276Z [qtp580718781-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=89be80fe-f40c-48b3-a440-f85197bc939d InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:36:52.275903Z InvocationID=1627ec82-554b-44d6-bc83-b9cd0c79369d LogTimestamp=2021-05-04T00:36:52.275969Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:36:52.277Z [qtp580718781-17] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=1 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:36:52.276Z InvocationID=6626ad94-fd1a-419b-ae78-163b426441b5 LogTimestamp=2021-05-04T00:36:52.277732Z ResponseDesc=OK Exiting. 2021-05-04T00:36:52.278Z [qtp580718781-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:36:53.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2182e93d-0acd-4f1c-b705-46f959e0122a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:53.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2182e93d-0acd-4f1c-b705-46f959e0122a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d1a10b1 2021-05-04T00:36:53.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2182e93d-0acd-4f1c-b705-46f959e0122a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:53.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2182e93d-0acd-4f1c-b705-46f959e0122a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:56.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eefcd758-8800-4356-84ac-d8f17f429676 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:56.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eefcd758-8800-4356-84ac-d8f17f429676 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55a7db34 2021-05-04T00:36:56.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eefcd758-8800-4356-84ac-d8f17f429676 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:56.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eefcd758-8800-4356-84ac-d8f17f429676 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:36:59.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc9220ba-af75-430c-b2eb-f29c294095b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:36:59.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc9220ba-af75-430c-b2eb-f29c294095b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21731a45 2021-05-04T00:36:59.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc9220ba-af75-430c-b2eb-f29c294095b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:36:59.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc9220ba-af75-430c-b2eb-f29c294095b4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:02.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=febd546d-2cdc-4447-9d29-c85d5f50aa5d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:02.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=febd546d-2cdc-4447-9d29-c85d5f50aa5d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73ce75b8 2021-05-04T00:37:02.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=febd546d-2cdc-4447-9d29-c85d5f50aa5d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:02.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=febd546d-2cdc-4447-9d29-c85d5f50aa5d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e76be92e-f319-474a-a341-13be47568b7d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e76be92e-f319-474a-a341-13be47568b7d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e8c7ce8 2021-05-04T00:37:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e76be92e-f319-474a-a341-13be47568b7d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e76be92e-f319-474a-a341-13be47568b7d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:08.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bde70726-de2d-4e68-9b7c-54af243f22e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:08.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bde70726-de2d-4e68-9b7c-54af243f22e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45996c76 2021-05-04T00:37:08.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bde70726-de2d-4e68-9b7c-54af243f22e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:08.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bde70726-de2d-4e68-9b7c-54af243f22e6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:11.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c384f578-721b-4802-a2e7-f624a50276f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:11.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c384f578-721b-4802-a2e7-f624a50276f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c01e7b2 2021-05-04T00:37:11.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c384f578-721b-4802-a2e7-f624a50276f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:11.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c384f578-721b-4802-a2e7-f624a50276f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:14.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2832536a-fdae-4cac-9439-96830eb0d60c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:14.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2832536a-fdae-4cac-9439-96830eb0d60c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c622e3f 2021-05-04T00:37:14.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2832536a-fdae-4cac-9439-96830eb0d60c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:14.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2832536a-fdae-4cac-9439-96830eb0d60c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:17.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61a530e1-4bbb-4638-bf99-1bca6c5dd0ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:17.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61a530e1-4bbb-4638-bf99-1bca6c5dd0ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31c46667 2021-05-04T00:37:17.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61a530e1-4bbb-4638-bf99-1bca6c5dd0ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:17.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61a530e1-4bbb-4638-bf99-1bca6c5dd0ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:20.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b24a7ba1-569a-4c10-8c0b-de8a1851d452 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:20.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b24a7ba1-569a-4c10-8c0b-de8a1851d452 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f3046f5 2021-05-04T00:37:20.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b24a7ba1-569a-4c10-8c0b-de8a1851d452 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:20.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b24a7ba1-569a-4c10-8c0b-de8a1851d452 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:23.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9f83a84-cd53-4277-8f2f-8d8cb067abd3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:23.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9f83a84-cd53-4277-8f2f-8d8cb067abd3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4cff0ad6 2021-05-04T00:37:23.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9f83a84-cd53-4277-8f2f-8d8cb067abd3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:23.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9f83a84-cd53-4277-8f2f-8d8cb067abd3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:25.881Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b868dbbc-6f8d-4afc-9590-20606ab07eb4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:25.881Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b868dbbc-6f8d-4afc-9590-20606ab07eb4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2101d6a0 2021-05-04T00:37:25.881Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b868dbbc-6f8d-4afc-9590-20606ab07eb4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:25.881Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b868dbbc-6f8d-4afc-9590-20606ab07eb4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:29.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ddf0d17d-561c-42ea-91b1-ab21e9d80f06 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:29.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ddf0d17d-561c-42ea-91b1-ab21e9d80f06 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33e47bb9 2021-05-04T00:37:29.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ddf0d17d-561c-42ea-91b1-ab21e9d80f06 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:29.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ddf0d17d-561c-42ea-91b1-ab21e9d80f06 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6208db78-f87e-421f-800c-3648710e5e88 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6208db78-f87e-421f-800c-3648710e5e88 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e5e94de 2021-05-04T00:37:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6208db78-f87e-421f-800c-3648710e5e88 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6208db78-f87e-421f-800c-3648710e5e88 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ddda92b-e7d8-47a8-bc5f-5aa6049e7a2d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ddda92b-e7d8-47a8-bc5f-5aa6049e7a2d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ac39679 2021-05-04T00:37:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ddda92b-e7d8-47a8-bc5f-5aa6049e7a2d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:35.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ddda92b-e7d8-47a8-bc5f-5aa6049e7a2d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:38.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=116fa304-25c4-4190-b056-f303a8017632 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:38.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=116fa304-25c4-4190-b056-f303a8017632 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1558e357 2021-05-04T00:37:38.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=116fa304-25c4-4190-b056-f303a8017632 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:38.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=116fa304-25c4-4190-b056-f303a8017632 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:40.955Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b55498ac-2189-4a45-88e2-9dc4a21fb165 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:40.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b55498ac-2189-4a45-88e2-9dc4a21fb165 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5717bf6 2021-05-04T00:37:40.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b55498ac-2189-4a45-88e2-9dc4a21fb165 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:40.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b55498ac-2189-4a45-88e2-9dc4a21fb165 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:44.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba6ef2e-ddd5-4469-a1a8-9964c961a433 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:44.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba6ef2e-ddd5-4469-a1a8-9964c961a433 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ff3f7e2 2021-05-04T00:37:44.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba6ef2e-ddd5-4469-a1a8-9964c961a433 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:44.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba6ef2e-ddd5-4469-a1a8-9964c961a433 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:47.349Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1a821a2-d7b3-4954-9d20-c0d4fe95529b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:47.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1a821a2-d7b3-4954-9d20-c0d4fe95529b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5925c63b 2021-05-04T00:37:47.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1a821a2-d7b3-4954-9d20-c0d4fe95529b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:47.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1a821a2-d7b3-4954-9d20-c0d4fe95529b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:50.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c65bb1d3-44a9-49d1-9df9-e18db73ff841 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:50.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c65bb1d3-44a9-49d1-9df9-e18db73ff841 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b35d413 2021-05-04T00:37:50.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c65bb1d3-44a9-49d1-9df9-e18db73ff841 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:50.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c65bb1d3-44a9-49d1-9df9-e18db73ff841 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff2c65d7-5577-4abe-b104-1ffa8286d55b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff2c65d7-5577-4abe-b104-1ffa8286d55b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@595b048 2021-05-04T00:37:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff2c65d7-5577-4abe-b104-1ffa8286d55b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff2c65d7-5577-4abe-b104-1ffa8286d55b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:56.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d39803ec-81ee-488b-b7a7-b79dd37eb68d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:56.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d39803ec-81ee-488b-b7a7-b79dd37eb68d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65b91bff 2021-05-04T00:37:56.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d39803ec-81ee-488b-b7a7-b79dd37eb68d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:56.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d39803ec-81ee-488b-b7a7-b79dd37eb68d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:37:59.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00e804b9-28dd-4b6a-ad77-3d6728ec8aff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:37:59.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00e804b9-28dd-4b6a-ad77-3d6728ec8aff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12da1d6d 2021-05-04T00:37:59.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00e804b9-28dd-4b6a-ad77-3d6728ec8aff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:37:59.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00e804b9-28dd-4b6a-ad77-3d6728ec8aff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:02.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=508e68c6-e2b9-492c-87f8-f2c32a08d7b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:02.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=508e68c6-e2b9-492c-87f8-f2c32a08d7b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37eac669 2021-05-04T00:38:02.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=508e68c6-e2b9-492c-87f8-f2c32a08d7b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:02.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=508e68c6-e2b9-492c-87f8-f2c32a08d7b8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1079ee5f-34bb-4f74-9bd4-440d782601ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1079ee5f-34bb-4f74-9bd4-440d782601ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f55372c 2021-05-04T00:38:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1079ee5f-34bb-4f74-9bd4-440d782601ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1079ee5f-34bb-4f74-9bd4-440d782601ca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:08.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de989467-cf5b-4f47-93a0-e0e2b0240e12 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:08.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de989467-cf5b-4f47-93a0-e0e2b0240e12 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@329eb1da 2021-05-04T00:38:08.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de989467-cf5b-4f47-93a0-e0e2b0240e12 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:08.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de989467-cf5b-4f47-93a0-e0e2b0240e12 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:11.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4251a5-4778-4db9-bf58-0fa322960ea7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4251a5-4778-4db9-bf58-0fa322960ea7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@706de391 2021-05-04T00:38:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4251a5-4778-4db9-bf58-0fa322960ea7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4251a5-4778-4db9-bf58-0fa322960ea7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:14.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05e817c1-5190-46bc-ba1b-c54bf21ee24d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:14.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05e817c1-5190-46bc-ba1b-c54bf21ee24d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1155ba7b 2021-05-04T00:38:14.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05e817c1-5190-46bc-ba1b-c54bf21ee24d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:14.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05e817c1-5190-46bc-ba1b-c54bf21ee24d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:17.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eb67839-582f-4048-beb4-349999e296cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:17.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eb67839-582f-4048-beb4-349999e296cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@270f132a 2021-05-04T00:38:17.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eb67839-582f-4048-beb4-349999e296cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:17.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eb67839-582f-4048-beb4-349999e296cd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:20.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8ce616b-cd26-4317-a610-18c7d08a101e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:20.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8ce616b-cd26-4317-a610-18c7d08a101e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60586670 2021-05-04T00:38:20.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8ce616b-cd26-4317-a610-18c7d08a101e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:20.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8ce616b-cd26-4317-a610-18c7d08a101e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:23.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03a982f-962c-4780-874c-f97bcd97eafe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:23.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03a982f-962c-4780-874c-f97bcd97eafe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f367081 2021-05-04T00:38:23.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03a982f-962c-4780-874c-f97bcd97eafe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:23.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03a982f-962c-4780-874c-f97bcd97eafe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dcadbfd-839e-49bc-8a20-67f5e3d29500 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dcadbfd-839e-49bc-8a20-67f5e3d29500 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@438c9b7e 2021-05-04T00:38:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dcadbfd-839e-49bc-8a20-67f5e3d29500 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dcadbfd-839e-49bc-8a20-67f5e3d29500 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:29.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94b147d7-b9d9-4bd1-8310-5fcbd8bde535 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:29.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94b147d7-b9d9-4bd1-8310-5fcbd8bde535 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c454666 2021-05-04T00:38:29.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94b147d7-b9d9-4bd1-8310-5fcbd8bde535 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:29.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94b147d7-b9d9-4bd1-8310-5fcbd8bde535 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:32.153Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0da12c13-5e33-46e1-a726-4b2f6920b9a9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:32.153Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0da12c13-5e33-46e1-a726-4b2f6920b9a9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f8dc878 2021-05-04T00:38:32.153Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0da12c13-5e33-46e1-a726-4b2f6920b9a9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:32.153Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0da12c13-5e33-46e1-a726-4b2f6920b9a9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:35.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c349262-26c2-4a38-b16b-d8ab3f988473 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:35.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c349262-26c2-4a38-b16b-d8ab3f988473 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73f31570 2021-05-04T00:38:35.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c349262-26c2-4a38-b16b-d8ab3f988473 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:35.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c349262-26c2-4a38-b16b-d8ab3f988473 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:38.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9887f53-7510-4dd1-8f20-e7dc1a7fc786 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:38.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9887f53-7510-4dd1-8f20-e7dc1a7fc786 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5483271b 2021-05-04T00:38:38.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9887f53-7510-4dd1-8f20-e7dc1a7fc786 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:38.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9887f53-7510-4dd1-8f20-e7dc1a7fc786 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=987472a7-67c8-4b3f-a91c-10d056c5b71a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=987472a7-67c8-4b3f-a91c-10d056c5b71a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62c32d9b 2021-05-04T00:38:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=987472a7-67c8-4b3f-a91c-10d056c5b71a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:41.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=987472a7-67c8-4b3f-a91c-10d056c5b71a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:44.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec700cab-f822-4fde-b577-eae1dffc3d78 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:44.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec700cab-f822-4fde-b577-eae1dffc3d78 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33dce7b3 2021-05-04T00:38:44.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec700cab-f822-4fde-b577-eae1dffc3d78 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:44.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec700cab-f822-4fde-b577-eae1dffc3d78 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:47.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4dcb78-9140-4165-80cf-a0a4eabb1f06 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:47.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4dcb78-9140-4165-80cf-a0a4eabb1f06 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@586f3c2d 2021-05-04T00:38:47.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4dcb78-9140-4165-80cf-a0a4eabb1f06 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:47.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4dcb78-9140-4165-80cf-a0a4eabb1f06 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:50.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d389056c-c73e-498d-a19f-6b4f36e90b41 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:50.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d389056c-c73e-498d-a19f-6b4f36e90b41 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@159973c4 2021-05-04T00:38:50.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d389056c-c73e-498d-a19f-6b4f36e90b41 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:50.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d389056c-c73e-498d-a19f-6b4f36e90b41 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:53.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06c6089a-0b68-4bfb-ad1f-a96b9040c235 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:53.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06c6089a-0b68-4bfb-ad1f-a96b9040c235 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e4d2b19 2021-05-04T00:38:53.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06c6089a-0b68-4bfb-ad1f-a96b9040c235 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:53.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06c6089a-0b68-4bfb-ad1f-a96b9040c235 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:56.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4c77075-bb58-4590-8bed-1d2fd31edb1b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:56.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4c77075-bb58-4590-8bed-1d2fd31edb1b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3af8b81c 2021-05-04T00:38:56.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4c77075-bb58-4590-8bed-1d2fd31edb1b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:56.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4c77075-bb58-4590-8bed-1d2fd31edb1b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:38:57.948Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=86d7d7e5-000d-45b0-bc9b-d01ac0ed9248 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:38:57.947497Z InvocationID=63824a63-3a71-4a7a-abf4-ee47d5af233d LogTimestamp=2021-05-04T00:38:57.947664Z Entering 2021-05-04T00:38:57.948Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=86d7d7e5-000d-45b0-bc9b-d01ac0ed9248 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:38:57.947497Z InvocationID=63824a63-3a71-4a7a-abf4-ee47d5af233d LogTimestamp=2021-05-04T00:38:57.947664Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:38:57.952Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:38:57.948Z InvocationID=d2a04bce-1cbe-4307-b047-16d4a8a769db LogTimestamp=2021-05-04T00:38:57.952113Z ResponseDesc=OK Exiting. 2021-05-04T00:38:57.952Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:38:59.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f81b5f38-2b2c-4bf9-ba7a-0dcd500c2383 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:38:59.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f81b5f38-2b2c-4bf9-ba7a-0dcd500c2383 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b57653a 2021-05-04T00:38:59.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f81b5f38-2b2c-4bf9-ba7a-0dcd500c2383 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:38:59.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f81b5f38-2b2c-4bf9-ba7a-0dcd500c2383 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:02.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6e00356-d313-4b16-96e8-e3981c2fe888 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:02.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6e00356-d313-4b16-96e8-e3981c2fe888 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@458e32c7 2021-05-04T00:39:02.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6e00356-d313-4b16-96e8-e3981c2fe888 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:02.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6e00356-d313-4b16-96e8-e3981c2fe888 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:05.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=480b7bda-ed39-4500-b90c-7f4242824a0d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:05.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=480b7bda-ed39-4500-b90c-7f4242824a0d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@429cdc8c 2021-05-04T00:39:05.613Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=480b7bda-ed39-4500-b90c-7f4242824a0d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:05.613Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=480b7bda-ed39-4500-b90c-7f4242824a0d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:08.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8e87ee4-fb8d-4c85-b57b-83df3da81a59 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:08.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8e87ee4-fb8d-4c85-b57b-83df3da81a59 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d5ba49b 2021-05-04T00:39:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8e87ee4-fb8d-4c85-b57b-83df3da81a59 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8e87ee4-fb8d-4c85-b57b-83df3da81a59 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adf2e769-5aaa-4866-bb20-a3f76c9d5211 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adf2e769-5aaa-4866-bb20-a3f76c9d5211 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d8568bf 2021-05-04T00:39:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adf2e769-5aaa-4866-bb20-a3f76c9d5211 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adf2e769-5aaa-4866-bb20-a3f76c9d5211 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:14.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68a1c694-b73c-441e-b2a4-88ab1df58d65 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:14.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68a1c694-b73c-441e-b2a4-88ab1df58d65 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@292969c2 2021-05-04T00:39:14.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68a1c694-b73c-441e-b2a4-88ab1df58d65 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:14.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68a1c694-b73c-441e-b2a4-88ab1df58d65 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:17.020Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51c8d9e8-987c-44b0-9645-dae0a506922a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:17.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51c8d9e8-987c-44b0-9645-dae0a506922a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d75a8d7 2021-05-04T00:39:17.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51c8d9e8-987c-44b0-9645-dae0a506922a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:17.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51c8d9e8-987c-44b0-9645-dae0a506922a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be35e695-9ead-44d3-93ac-ae06273a5541 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be35e695-9ead-44d3-93ac-ae06273a5541 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6deaaa9e 2021-05-04T00:39:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be35e695-9ead-44d3-93ac-ae06273a5541 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be35e695-9ead-44d3-93ac-ae06273a5541 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:23.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=000d695a-492c-4d8e-8e3f-224dc7f85f29 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:23.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=000d695a-492c-4d8e-8e3f-224dc7f85f29 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b9ce70c 2021-05-04T00:39:23.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=000d695a-492c-4d8e-8e3f-224dc7f85f29 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:23.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=000d695a-492c-4d8e-8e3f-224dc7f85f29 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:26.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c84139ad-f1be-4700-84e9-711cfa6f051b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:26.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c84139ad-f1be-4700-84e9-711cfa6f051b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3999f2c4 2021-05-04T00:39:26.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c84139ad-f1be-4700-84e9-711cfa6f051b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:26.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c84139ad-f1be-4700-84e9-711cfa6f051b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:29.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112c730c-7bcb-42f4-b0d5-31997b594052 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:29.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112c730c-7bcb-42f4-b0d5-31997b594052 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@190ad630 2021-05-04T00:39:29.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112c730c-7bcb-42f4-b0d5-31997b594052 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:29.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112c730c-7bcb-42f4-b0d5-31997b594052 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:32.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7657d30f-1723-4c62-8299-6fe02af72279 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:32.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7657d30f-1723-4c62-8299-6fe02af72279 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@210c21d5 2021-05-04T00:39:32.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7657d30f-1723-4c62-8299-6fe02af72279 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:32.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7657d30f-1723-4c62-8299-6fe02af72279 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:35.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7bcfb922-8245-47ff-a4c0-ef96b98eda1a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:35.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7bcfb922-8245-47ff-a4c0-ef96b98eda1a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d14067f 2021-05-04T00:39:35.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7bcfb922-8245-47ff-a4c0-ef96b98eda1a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:35.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7bcfb922-8245-47ff-a4c0-ef96b98eda1a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:38.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db52e14-1e0e-4be3-bdc6-9e774f143afc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:38.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db52e14-1e0e-4be3-bdc6-9e774f143afc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6bedc8cb 2021-05-04T00:39:38.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db52e14-1e0e-4be3-bdc6-9e774f143afc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:38.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db52e14-1e0e-4be3-bdc6-9e774f143afc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:41.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e50ab586-8afe-4702-b662-4d9e4125bc18 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:41.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e50ab586-8afe-4702-b662-4d9e4125bc18 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@249928d0 2021-05-04T00:39:41.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e50ab586-8afe-4702-b662-4d9e4125bc18 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:41.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e50ab586-8afe-4702-b662-4d9e4125bc18 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:44.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=421d6235-5571-477e-b0be-d878448ad00c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:44.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=421d6235-5571-477e-b0be-d878448ad00c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64dbb71f 2021-05-04T00:39:44.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=421d6235-5571-477e-b0be-d878448ad00c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:44.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=421d6235-5571-477e-b0be-d878448ad00c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:47.086Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=774d1f3f-8dea-4d95-a3fa-4162dee37947 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:47.086Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=774d1f3f-8dea-4d95-a3fa-4162dee37947 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59c17205 2021-05-04T00:39:47.087Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=774d1f3f-8dea-4d95-a3fa-4162dee37947 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:47.087Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=774d1f3f-8dea-4d95-a3fa-4162dee37947 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:50.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=85a9af8c-4c4c-4621-a4b2-7b41746afbce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:50.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=85a9af8c-4c4c-4621-a4b2-7b41746afbce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c6f5266 2021-05-04T00:39:50.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=85a9af8c-4c4c-4621-a4b2-7b41746afbce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:50.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=85a9af8c-4c4c-4621-a4b2-7b41746afbce PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:53.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d3fb464-0c3a-4260-a7b6-6903d0c6e2aa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:53.548Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d3fb464-0c3a-4260-a7b6-6903d0c6e2aa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a041eb7 2021-05-04T00:39:53.548Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d3fb464-0c3a-4260-a7b6-6903d0c6e2aa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:53.548Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d3fb464-0c3a-4260-a7b6-6903d0c6e2aa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:56.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9980e02-f140-45c3-b7c5-ae10e1b36086 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:56.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9980e02-f140-45c3-b7c5-ae10e1b36086 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c5e8bf3 2021-05-04T00:39:56.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9980e02-f140-45c3-b7c5-ae10e1b36086 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:56.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9980e02-f140-45c3-b7c5-ae10e1b36086 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:39:59.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01306dcb-7a42-4d5b-9e4b-38c48154c93d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:39:59.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01306dcb-7a42-4d5b-9e4b-38c48154c93d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@242f1b4d 2021-05-04T00:39:59.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01306dcb-7a42-4d5b-9e4b-38c48154c93d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:39:59.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01306dcb-7a42-4d5b-9e4b-38c48154c93d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:02.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fa3b1db-938c-4200-874b-b0f8cd014d5b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:02.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fa3b1db-938c-4200-874b-b0f8cd014d5b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@143f6bf4 2021-05-04T00:40:02.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fa3b1db-938c-4200-874b-b0f8cd014d5b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:02.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fa3b1db-938c-4200-874b-b0f8cd014d5b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:05.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4770836f-2108-4f08-8524-ca5d2181336c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:05.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4770836f-2108-4f08-8524-ca5d2181336c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6eb43e62 2021-05-04T00:40:05.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4770836f-2108-4f08-8524-ca5d2181336c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:05.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4770836f-2108-4f08-8524-ca5d2181336c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:08.066Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01kdi3o80zktof6j8xhcqypi8x245 2021-05-04T00:40:08.066Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01pg7r4kuvngn55lvw86j6xai1244 2021-05-04T00:40:08.066Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01jeuj9xfsgtdtp53yf6a3rwvi247 2021-05-04T00:40:08.066Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0qvqx7gppqzga1cs17ktz8b5bm249 2021-05-04T00:40:08.066Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node08ds0qtyy4s8j15blzs93a1hgq254 2021-05-04T00:40:08.067Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01khsnbhajj5j21mzgzliper0bd252 2021-05-04T00:40:08.067Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0gf1d8w177lpzgrmt3u61d12i253 2021-05-04T00:40:08.067Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01qhy8sciaqt4hylnrmp9cn3v250 2021-05-04T00:40:08.067Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01x5guliu7xwdq9p1j6mheujzn255 2021-05-04T00:40:08.067Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01jvy6ogmfrwkdjph1skq18wjs248 2021-05-04T00:40:08.067Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node037pjj3axbx8p1atbnto43bhhf256 2021-05-04T00:40:08.067Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0wwgonzhd3z8f12dc8whqq0o4r251 2021-05-04T00:40:08.068Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node013e0t5ug1eihq1fb2pc8iylf9b246 2021-05-04T00:40:08.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1de3e3ae-9b21-4b3e-9e43-a8c9859c0c16 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:08.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1de3e3ae-9b21-4b3e-9e43-a8c9859c0c16 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66bd2154 2021-05-04T00:40:08.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1de3e3ae-9b21-4b3e-9e43-a8c9859c0c16 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:08.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1de3e3ae-9b21-4b3e-9e43-a8c9859c0c16 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:11.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0ae8cc6-71cf-4873-ad4f-f6bff796feec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:11.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0ae8cc6-71cf-4873-ad4f-f6bff796feec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a94abbd 2021-05-04T00:40:11.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0ae8cc6-71cf-4873-ad4f-f6bff796feec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:11.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0ae8cc6-71cf-4873-ad4f-f6bff796feec PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=492b1cd3-a5ec-499e-8cc3-3126d315f080 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=492b1cd3-a5ec-499e-8cc3-3126d315f080 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@551b17d0 2021-05-04T00:40:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=492b1cd3-a5ec-499e-8cc3-3126d315f080 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=492b1cd3-a5ec-499e-8cc3-3126d315f080 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b64a393d-3b1b-4137-93d7-6a471c23ab8d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b64a393d-3b1b-4137-93d7-6a471c23ab8d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a93743f 2021-05-04T00:40:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b64a393d-3b1b-4137-93d7-6a471c23ab8d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b64a393d-3b1b-4137-93d7-6a471c23ab8d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:20.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49a2d4ea-60d6-43a8-824d-584f4dfdc1bc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:20.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49a2d4ea-60d6-43a8-824d-584f4dfdc1bc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15ab51b8 2021-05-04T00:40:20.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49a2d4ea-60d6-43a8-824d-584f4dfdc1bc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:20.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49a2d4ea-60d6-43a8-824d-584f4dfdc1bc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:23.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5d41012-5ff6-429e-b5de-4f8781bd3259 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:23.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5d41012-5ff6-429e-b5de-4f8781bd3259 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a7dd9f3 2021-05-04T00:40:23.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5d41012-5ff6-429e-b5de-4f8781bd3259 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:23.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5d41012-5ff6-429e-b5de-4f8781bd3259 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:25.977Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=459a4ab2-a90b-41a1-8e30-3ca3ff2a6191 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:25.977Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=459a4ab2-a90b-41a1-8e30-3ca3ff2a6191 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@755eafb 2021-05-04T00:40:25.977Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=459a4ab2-a90b-41a1-8e30-3ca3ff2a6191 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:25.977Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=459a4ab2-a90b-41a1-8e30-3ca3ff2a6191 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a7954b0-4221-4dd4-817d-ac8c5a17a150 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a7954b0-4221-4dd4-817d-ac8c5a17a150 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66644a2 2021-05-04T00:40:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a7954b0-4221-4dd4-817d-ac8c5a17a150 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a7954b0-4221-4dd4-817d-ac8c5a17a150 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:32.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=529f1729-091f-4b0b-83c7-3a4b07f8f649 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:32.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=529f1729-091f-4b0b-83c7-3a4b07f8f649 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@490fe347 2021-05-04T00:40:32.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=529f1729-091f-4b0b-83c7-3a4b07f8f649 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:32.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=529f1729-091f-4b0b-83c7-3a4b07f8f649 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8796c2f8-e4e5-4743-b21f-e3c19e56f562 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8796c2f8-e4e5-4743-b21f-e3c19e56f562 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d5909d4 2021-05-04T00:40:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8796c2f8-e4e5-4743-b21f-e3c19e56f562 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8796c2f8-e4e5-4743-b21f-e3c19e56f562 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:36.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:36.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.304Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.305Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e21d29f3-18bd-4404-8080-c6c50bbbf670 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:38.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e21d29f3-18bd-4404-8080-c6c50bbbf670 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5733cb6 2021-05-04T00:40:38.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e21d29f3-18bd-4404-8080-c6c50bbbf670 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:38.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e21d29f3-18bd-4404-8080-c6c50bbbf670 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:38.379Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.380Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.386Z [qtp580718781-16] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=487d7784-31b6-406c-a77f-bb6cffbbc2ae InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:40:38.385781Z InvocationID=7085ef60-dfff-4adf-9246-42f16c5c5ed1 LogTimestamp=2021-05-04T00:40:38.385882Z Entering 2021-05-04T00:40:38.386Z [qtp580718781-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=487d7784-31b6-406c-a77f-bb6cffbbc2ae InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:40:38.385781Z InvocationID=7085ef60-dfff-4adf-9246-42f16c5c5ed1 LogTimestamp=2021-05-04T00:40:38.385882Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:40:38.391Z [qtp580718781-16] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.386Z InvocationID=617914d4-fbad-4908-9e6b-47bf567be8b4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/ePRuBsL1rqKW0SVf will send credentials over a clear channel. 2021-05-04T00:40:38.391Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.386Z InvocationID=617914d4-fbad-4908-9e6b-47bf567be8b4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/ePRuBsL1rqKW0SVf (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:40:38.395Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.395Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.414Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.415Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.421Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=62ce7c14-7a1c-4937-a996-180916cd0208 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:40:38.421439Z InvocationID=f5ef222e-b4d6-4a8c-adc1-39e4ddff5d6a LogTimestamp=2021-05-04T00:40:38.421478Z Entering 2021-05-04T00:40:38.421Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=62ce7c14-7a1c-4937-a996-180916cd0208 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:40:38.421439Z InvocationID=f5ef222e-b4d6-4a8c-adc1-39e4ddff5d6a LogTimestamp=2021-05-04T00:40:38.421478Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:40:38.426Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.421Z InvocationID=ab90fa5e-3edd-497f-9e4f-c6cd74cfdefa PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/EfXOZsENssQ1OnyB will send credentials over a clear channel. 2021-05-04T00:40:38.426Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.421Z InvocationID=ab90fa5e-3edd-497f-9e4f-c6cd74cfdefa PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/EfXOZsENssQ1OnyB (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:40:38.438Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.386Z InvocationID=617914d4-fbad-4908-9e6b-47bf567be8b4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.438Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.439Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.446Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.421Z InvocationID=ab90fa5e-3edd-497f-9e4f-c6cd74cfdefa PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.446Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.447Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.449Z [qtp580718781-16] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "ePRuBsL1rqKW0SVf" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.386Z InvocationID=617914d4-fbad-4908-9e6b-47bf567be8b4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/ePRuBsL1rqKW0SVf will send credentials over a clear channel. 2021-05-04T00:40:38.449Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "ePRuBsL1rqKW0SVf" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.386Z InvocationID=617914d4-fbad-4908-9e6b-47bf567be8b4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/ePRuBsL1rqKW0SVf (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:40:38.450Z [qtp580718781-12] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "EfXOZsENssQ1OnyB" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.421Z InvocationID=ab90fa5e-3edd-497f-9e4f-c6cd74cfdefa PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/EfXOZsENssQ1OnyB will send credentials over a clear channel. 2021-05-04T00:40:38.451Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "EfXOZsENssQ1OnyB" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.421Z InvocationID=ab90fa5e-3edd-497f-9e4f-c6cd74cfdefa PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/EfXOZsENssQ1OnyB (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:40:38.451Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.451Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.456Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.457Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.460Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "ePRuBsL1rqKW0SVf" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.386Z InvocationID=617914d4-fbad-4908-9e6b-47bf567be8b4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.461Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.462Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.469Z [qtp580718781-12] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "EfXOZsENssQ1OnyB" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.421Z InvocationID=ab90fa5e-3edd-497f-9e4f-c6cd74cfdefa PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.470Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.470Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.474Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.475Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.499Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "cds" API_KEY = "EfXOZsENssQ1OnyB" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=78 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.421Z InvocationID=ab90fa5e-3edd-497f-9e4f-c6cd74cfdefa LogTimestamp=2021-05-04T00:40:38.499275Z ResponseDesc=OK Exiting. 2021-05-04T00:40:38.499Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:40:38.500Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.501Z [qtp580718781-16] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "ves-openapi-manager" API_KEY = "ePRuBsL1rqKW0SVf" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=114 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:40:38.386Z InvocationID=617914d4-fbad-4908-9e6b-47bf567be8b4 LogTimestamp=2021-05-04T00:40:38.500886Z ResponseDesc=OK Exiting. 2021-05-04T00:40:38.501Z [qtp580718781-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:40:38.501Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.509Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.510Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.520Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.521Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.527Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.528Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.540Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.541Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.569Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.570Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.623Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.624Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.648Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.649Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.664Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.666Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.700Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.702Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.732Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.733Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.744Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.745Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.764Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.765Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.783Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.784Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.802Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.803Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.822Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.823Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.835Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.836Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.852Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.853Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.868Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.869Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.876Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.877Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.884Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.885Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.895Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.896Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.908Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.909Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:38.933Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:38.934Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.061Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.062Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.068Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.069Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.117Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.118Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.130Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.131Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.135Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.136Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.140Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.141Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.152Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.153Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.162Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.164Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.173Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.187Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.188Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.200Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.201Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.205Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.206Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.213Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.214Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.218Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.219Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.222Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.225Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.234Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.235Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:39.240Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:39.240Z [UEB-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend connection recovery to UEB Cluster. 2021-05-04T00:40:41.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6d2fbe4-0a63-4bc2-872f-acd9527e9ce9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:41.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6d2fbe4-0a63-4bc2-872f-acd9527e9ce9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31842bee 2021-05-04T00:40:41.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6d2fbe4-0a63-4bc2-872f-acd9527e9ce9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:41.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6d2fbe4-0a63-4bc2-872f-acd9527e9ce9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:41.512Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b6d2fbe4-0a63-4bc2-872f-acd9527e9ce9 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:40:41.512Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b6d2fbe4-0a63-4bc2-872f-acd9527e9ce9 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:40:44.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbe7dd23-7a02-4aff-b197-314b37752c0d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:44.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbe7dd23-7a02-4aff-b197-314b37752c0d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@727f342 2021-05-04T00:40:44.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbe7dd23-7a02-4aff-b197-314b37752c0d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:44.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbe7dd23-7a02-4aff-b197-314b37752c0d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:47.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50792515-53ec-4d5d-bbb8-3dca6a0aa924 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:47.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50792515-53ec-4d5d-bbb8-3dca6a0aa924 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62d54ddb 2021-05-04T00:40:47.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50792515-53ec-4d5d-bbb8-3dca6a0aa924 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:47.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50792515-53ec-4d5d-bbb8-3dca6a0aa924 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:40:50.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:40:50.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba1082a-9f48-4e64-b7f5-4556faadfe3d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:50.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba1082a-9f48-4e64-b7f5-4556faadfe3d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7edf4bbb 2021-05-04T00:40:50.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba1082a-9f48-4e64-b7f5-4556faadfe3d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:50.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba1082a-9f48-4e64-b7f5-4556faadfe3d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:53.332Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:40:53.333Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:40:53.334Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:40:53.334Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:40:53.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=892b8c18-b7cb-4f01-b5e0-03c0a5008994 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:53.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=892b8c18-b7cb-4f01-b5e0-03c0a5008994 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e94c4d5 2021-05-04T00:40:53.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=892b8c18-b7cb-4f01-b5e0-03c0a5008994 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:53.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=892b8c18-b7cb-4f01-b5e0-03c0a5008994 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:56.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88e70887-4fc7-4ec0-bcee-7b51d2f98642 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:56.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88e70887-4fc7-4ec0-bcee-7b51d2f98642 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@561187a6 2021-05-04T00:40:56.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88e70887-4fc7-4ec0-bcee-7b51d2f98642 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:56.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88e70887-4fc7-4ec0-bcee-7b51d2f98642 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:40:59.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2aeb2e6c-ab76-46f0-ba4c-1a77ac2a774f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:40:59.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2aeb2e6c-ab76-46f0-ba4c-1a77ac2a774f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ee2b32a 2021-05-04T00:40:59.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2aeb2e6c-ab76-46f0-ba4c-1a77ac2a774f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:40:59.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2aeb2e6c-ab76-46f0-ba4c-1a77ac2a774f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:02.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11ceb41e-483e-4237-b453-f812f8bf7068 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:02.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11ceb41e-483e-4237-b453-f812f8bf7068 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ae4b1de 2021-05-04T00:41:02.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11ceb41e-483e-4237-b453-f812f8bf7068 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:02.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11ceb41e-483e-4237-b453-f812f8bf7068 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:41:05.212Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:41:05.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ea6c9a6-fbe5-4f32-af3b-54178b9073b1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:05.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ea6c9a6-fbe5-4f32-af3b-54178b9073b1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@521e6c3b 2021-05-04T00:41:05.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ea6c9a6-fbe5-4f32-af3b-54178b9073b1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:05.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ea6c9a6-fbe5-4f32-af3b-54178b9073b1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:08.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74865d04-1c16-47dd-a27c-39d1e1774896 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:08.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74865d04-1c16-47dd-a27c-39d1e1774896 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@868fd69 2021-05-04T00:41:08.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74865d04-1c16-47dd-a27c-39d1e1774896 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:08.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74865d04-1c16-47dd-a27c-39d1e1774896 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:08.302Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:41:08.303Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:41:08.304Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:41:08.304Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:41:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f4ac7c0-456d-4b8d-a867-4e5edf2cfc5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f4ac7c0-456d-4b8d-a867-4e5edf2cfc5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39864ad 2021-05-04T00:41:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f4ac7c0-456d-4b8d-a867-4e5edf2cfc5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:11.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f4ac7c0-456d-4b8d-a867-4e5edf2cfc5f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:14.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99725244-bf56-4491-a251-09f11ec18f72 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:14.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99725244-bf56-4491-a251-09f11ec18f72 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ccd5f41 2021-05-04T00:41:14.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99725244-bf56-4491-a251-09f11ec18f72 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:14.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99725244-bf56-4491-a251-09f11ec18f72 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:17.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=185cfc27-b7a5-402f-b714-d9c1782ca441 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:17.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=185cfc27-b7a5-402f-b714-d9c1782ca441 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d138357 2021-05-04T00:41:17.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=185cfc27-b7a5-402f-b714-d9c1782ca441 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:17.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=185cfc27-b7a5-402f-b714-d9c1782ca441 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:41:20.192Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:41:20.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a14da8d6-ddca-4f73-aa0e-8c48c17ec8a8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:20.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a14da8d6-ddca-4f73-aa0e-8c48c17ec8a8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52dfb629 2021-05-04T00:41:20.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a14da8d6-ddca-4f73-aa0e-8c48c17ec8a8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:20.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a14da8d6-ddca-4f73-aa0e-8c48c17ec8a8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:23.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:41:23.369Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:41:23.370Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:41:23.370Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:41:23.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456c1e4b-f9df-41c2-b538-4eaf4871b894 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:23.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456c1e4b-f9df-41c2-b538-4eaf4871b894 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12220dd3 2021-05-04T00:41:23.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456c1e4b-f9df-41c2-b538-4eaf4871b894 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:23.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456c1e4b-f9df-41c2-b538-4eaf4871b894 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:26.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f64ca01-77db-4cc0-9cd1-18ffa4c89d14 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:26.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f64ca01-77db-4cc0-9cd1-18ffa4c89d14 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44c36f81 2021-05-04T00:41:26.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f64ca01-77db-4cc0-9cd1-18ffa4c89d14 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:26.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f64ca01-77db-4cc0-9cd1-18ffa4c89d14 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:29.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78e3144d-f5d5-4277-be18-a7521e02ea1d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:29.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78e3144d-f5d5-4277-be18-a7521e02ea1d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@454b4575 2021-05-04T00:41:29.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78e3144d-f5d5-4277-be18-a7521e02ea1d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:29.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78e3144d-f5d5-4277-be18-a7521e02ea1d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:32.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1de79ac-98ea-4c1e-825f-ba21b619bcf8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:32.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1de79ac-98ea-4c1e-825f-ba21b619bcf8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a8add0b 2021-05-04T00:41:32.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1de79ac-98ea-4c1e-825f-ba21b619bcf8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:32.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1de79ac-98ea-4c1e-825f-ba21b619bcf8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:41:35.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fa86ce8-ef65-44b1-b9d1-bb021e76d5d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:35.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fa86ce8-ef65-44b1-b9d1-bb021e76d5d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68c4191c 2021-05-04T00:41:35.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fa86ce8-ef65-44b1-b9d1-bb021e76d5d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:35.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fa86ce8-ef65-44b1-b9d1-bb021e76d5d9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:38.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=277e0702-ef14-4d14-9e1b-03397ca4583a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:38.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=277e0702-ef14-4d14-9e1b-03397ca4583a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37a4dce2 2021-05-04T00:41:38.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=277e0702-ef14-4d14-9e1b-03397ca4583a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:38.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=277e0702-ef14-4d14-9e1b-03397ca4583a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:38.864Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:41:39.830Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:41:39.830Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:41:39.831Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:41:39.831Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:41:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df8e8f7d-e887-4e03-8ae8-4b5d56dc02e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df8e8f7d-e887-4e03-8ae8-4b5d56dc02e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28db578e 2021-05-04T00:41:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df8e8f7d-e887-4e03-8ae8-4b5d56dc02e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:41.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df8e8f7d-e887-4e03-8ae8-4b5d56dc02e0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:44.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da5c7829-a238-4637-a915-ba80f1390d37 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:44.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da5c7829-a238-4637-a915-ba80f1390d37 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1405b51b 2021-05-04T00:41:44.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da5c7829-a238-4637-a915-ba80f1390d37 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:44.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da5c7829-a238-4637-a915-ba80f1390d37 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:46.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bc74ce1-1d8c-4f90-8673-8dcc5cc1b624 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:46.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bc74ce1-1d8c-4f90-8673-8dcc5cc1b624 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5cc1527e 2021-05-04T00:41:46.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bc74ce1-1d8c-4f90-8673-8dcc5cc1b624 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:46.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bc74ce1-1d8c-4f90-8673-8dcc5cc1b624 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:41:50.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:41:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8595272-2956-4243-a109-62b816914f21 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8595272-2956-4243-a109-62b816914f21 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d93d6c8 2021-05-04T00:41:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8595272-2956-4243-a109-62b816914f21 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8595272-2956-4243-a109-62b816914f21 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:53.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68c6c0b9-43bf-456c-9bfd-368373b25b96 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:53.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68c6c0b9-43bf-456c-9bfd-368373b25b96 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f069fd3 2021-05-04T00:41:53.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68c6c0b9-43bf-456c-9bfd-368373b25b96 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:53.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68c6c0b9-43bf-456c-9bfd-368373b25b96 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:54.910Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:41:54.910Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:41:54.911Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:41:54.911Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:41:56.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4456e8e-de72-400f-998d-866fa0e25a78 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:56.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4456e8e-de72-400f-998d-866fa0e25a78 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6124845 2021-05-04T00:41:56.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4456e8e-de72-400f-998d-866fa0e25a78 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:56.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4456e8e-de72-400f-998d-866fa0e25a78 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:41:59.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2ba2232-e83b-4792-99e6-48b8813e9335 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:41:59.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2ba2232-e83b-4792-99e6-48b8813e9335 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5016b85b 2021-05-04T00:41:59.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2ba2232-e83b-4792-99e6-48b8813e9335 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:41:59.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2ba2232-e83b-4792-99e6-48b8813e9335 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:02.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12a73754-28e5-47d6-8450-1669f0f7fa6c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:02.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12a73754-28e5-47d6-8450-1669f0f7fa6c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e00107 2021-05-04T00:42:02.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12a73754-28e5-47d6-8450-1669f0f7fa6c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:02.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12a73754-28e5-47d6-8450-1669f0f7fa6c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:42:05.182Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:42:05.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4779c604-5653-4e8b-8446-eb7f86b01079 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:05.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4779c604-5653-4e8b-8446-eb7f86b01079 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@893eeaf 2021-05-04T00:42:05.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4779c604-5653-4e8b-8446-eb7f86b01079 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4779c604-5653-4e8b-8446-eb7f86b01079 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:08.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=147faf7b-d55a-4d50-bac6-b798d36c4108 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:08.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=147faf7b-d55a-4d50-bac6-b798d36c4108 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c57df98 2021-05-04T00:42:08.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=147faf7b-d55a-4d50-bac6-b798d36c4108 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:08.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=147faf7b-d55a-4d50-bac6-b798d36c4108 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:09.854Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:42:09.855Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:42:09.855Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:42:09.855Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:42:11.121Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1b8d7e5-5260-41de-ae4e-7626ccd7c8e8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:11.121Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1b8d7e5-5260-41de-ae4e-7626ccd7c8e8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79667829 2021-05-04T00:42:11.121Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1b8d7e5-5260-41de-ae4e-7626ccd7c8e8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:11.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1b8d7e5-5260-41de-ae4e-7626ccd7c8e8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:14.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19bb3ae8-a624-42a3-b318-2a449407e522 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:14.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19bb3ae8-a624-42a3-b318-2a449407e522 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a0da63b 2021-05-04T00:42:14.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19bb3ae8-a624-42a3-b318-2a449407e522 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:14.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19bb3ae8-a624-42a3-b318-2a449407e522 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:17.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfce6b56-51c1-471b-abd4-84c3715dfdc9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:17.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfce6b56-51c1-471b-abd4-84c3715dfdc9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d7cc943 2021-05-04T00:42:17.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfce6b56-51c1-471b-abd4-84c3715dfdc9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:17.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfce6b56-51c1-471b-abd4-84c3715dfdc9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:20.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ce85f91-cf31-44dc-b3d4-9a5a434497f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:20.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ce85f91-cf31-44dc-b3d4-9a5a434497f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ff5706b 2021-05-04T00:42:20.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ce85f91-cf31-44dc-b3d4-9a5a434497f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:20.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ce85f91-cf31-44dc-b3d4-9a5a434497f1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:42:20.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:42:23.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28d668ff-4a4e-46db-a497-92ea04e76563 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:23.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28d668ff-4a4e-46db-a497-92ea04e76563 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@518ee8c 2021-05-04T00:42:23.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28d668ff-4a4e-46db-a497-92ea04e76563 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:23.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28d668ff-4a4e-46db-a497-92ea04e76563 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:24.791Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:42:24.792Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:42:24.793Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:42:24.793Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:42:26.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4023a0d9-4a95-418b-97f1-b4e9bccea709 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:26.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4023a0d9-4a95-418b-97f1-b4e9bccea709 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12b3ac6a 2021-05-04T00:42:26.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4023a0d9-4a95-418b-97f1-b4e9bccea709 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:26.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4023a0d9-4a95-418b-97f1-b4e9bccea709 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:29.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=592d655b-fedc-4d3a-8350-9e3b5ea71c95 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:29.018Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=592d655b-fedc-4d3a-8350-9e3b5ea71c95 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@190e59ae 2021-05-04T00:42:29.018Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=592d655b-fedc-4d3a-8350-9e3b5ea71c95 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:29.018Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=592d655b-fedc-4d3a-8350-9e3b5ea71c95 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:32.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06229fbb-099a-4c60-99fd-5831951a7fd9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:32.249Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06229fbb-099a-4c60-99fd-5831951a7fd9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7d9c04cc 2021-05-04T00:42:32.249Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06229fbb-099a-4c60-99fd-5831951a7fd9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:32.249Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06229fbb-099a-4c60-99fd-5831951a7fd9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:42:35.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:42:35.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b55e8a-c39a-44fe-b9c4-a75f7a3275ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:35.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b55e8a-c39a-44fe-b9c4-a75f7a3275ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f32f97d 2021-05-04T00:42:35.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b55e8a-c39a-44fe-b9c4-a75f7a3275ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:35.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b55e8a-c39a-44fe-b9c4-a75f7a3275ba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:38.234Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e9ee71c-f016-4e00-ab0f-f58e10f0554f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:38.234Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e9ee71c-f016-4e00-ab0f-f58e10f0554f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30edd20d 2021-05-04T00:42:38.234Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e9ee71c-f016-4e00-ab0f-f58e10f0554f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:38.234Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e9ee71c-f016-4e00-ab0f-f58e10f0554f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:39.789Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:42:39.790Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:42:39.790Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:42:39.790Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:42:41.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d16fad2c-8f07-4ef2-9019-068aefc4ca04 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:41.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d16fad2c-8f07-4ef2-9019-068aefc4ca04 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b453e97 2021-05-04T00:42:41.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d16fad2c-8f07-4ef2-9019-068aefc4ca04 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:41.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d16fad2c-8f07-4ef2-9019-068aefc4ca04 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ec75772-ee99-4d4d-ab75-53fbaf2604cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ec75772-ee99-4d4d-ab75-53fbaf2604cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46e2eb57 2021-05-04T00:42:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ec75772-ee99-4d4d-ab75-53fbaf2604cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ec75772-ee99-4d4d-ab75-53fbaf2604cb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:47.075Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3daf0e24-2896-4f09-904f-ee002d3d8304 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:47.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3daf0e24-2896-4f09-904f-ee002d3d8304 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6eb30be7 2021-05-04T00:42:47.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3daf0e24-2896-4f09-904f-ee002d3d8304 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:47.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3daf0e24-2896-4f09-904f-ee002d3d8304 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:42:50.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:42:50.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afc7c358-b746-40c5-a1f3-e9d2b37cbd04 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:50.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afc7c358-b746-40c5-a1f3-e9d2b37cbd04 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3090d186 2021-05-04T00:42:50.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afc7c358-b746-40c5-a1f3-e9d2b37cbd04 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:50.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afc7c358-b746-40c5-a1f3-e9d2b37cbd04 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:53.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67d6b9c2-5c55-4f81-897a-d61d4b76c41c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:53.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67d6b9c2-5c55-4f81-897a-d61d4b76c41c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4253f085 2021-05-04T00:42:53.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67d6b9c2-5c55-4f81-897a-d61d4b76c41c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:53.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67d6b9c2-5c55-4f81-897a-d61d4b76c41c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:54.729Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:42:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee6f1846-d5ae-4a3c-aa26-306be9511092 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee6f1846-d5ae-4a3c-aa26-306be9511092 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13ba03cb 2021-05-04T00:42:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee6f1846-d5ae-4a3c-aa26-306be9511092 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:56.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee6f1846-d5ae-4a3c-aa26-306be9511092 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:42:59.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76af308b-bb90-4574-949c-1c4c7aaebec9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:42:59.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76af308b-bb90-4574-949c-1c4c7aaebec9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6520ce68 2021-05-04T00:42:59.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76af308b-bb90-4574-949c-1c4c7aaebec9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:42:59.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76af308b-bb90-4574-949c-1c4c7aaebec9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:02.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc402cfd-52ff-48a7-8f50-531c4607318e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc402cfd-52ff-48a7-8f50-531c4607318e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@530a2fab 2021-05-04T00:43:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc402cfd-52ff-48a7-8f50-531c4607318e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc402cfd-52ff-48a7-8f50-531c4607318e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:43:05.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:43:05.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7002862-0012-4465-93f4-a10761dd4864 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:05.523Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7002862-0012-4465-93f4-a10761dd4864 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e803a97 2021-05-04T00:43:05.523Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7002862-0012-4465-93f4-a10761dd4864 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:05.523Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7002862-0012-4465-93f4-a10761dd4864 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c69d594d-1e4a-4058-93c2-80fa2c119770 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c69d594d-1e4a-4058-93c2-80fa2c119770 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d67d4d7 2021-05-04T00:43:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c69d594d-1e4a-4058-93c2-80fa2c119770 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c69d594d-1e4a-4058-93c2-80fa2c119770 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:11.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93f19b4c-cc80-4db0-8ddc-6d225035eebc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:11.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93f19b4c-cc80-4db0-8ddc-6d225035eebc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ed7109d 2021-05-04T00:43:11.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93f19b4c-cc80-4db0-8ddc-6d225035eebc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:11.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93f19b4c-cc80-4db0-8ddc-6d225035eebc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62371db2-07c2-4da7-a1ff-8a6bbe217293 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62371db2-07c2-4da7-a1ff-8a6bbe217293 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c22159b 2021-05-04T00:43:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62371db2-07c2-4da7-a1ff-8a6bbe217293 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62371db2-07c2-4da7-a1ff-8a6bbe217293 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:17.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d010f22e-df63-4635-aaf0-113ec77b6ce5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:17.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d010f22e-df63-4635-aaf0-113ec77b6ce5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23f9d60a 2021-05-04T00:43:17.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d010f22e-df63-4635-aaf0-113ec77b6ce5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:17.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d010f22e-df63-4635-aaf0-113ec77b6ce5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:43:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:43:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f067114-c66a-47eb-a956-4f9346951de7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f067114-c66a-47eb-a956-4f9346951de7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50cb7fce 2021-05-04T00:43:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f067114-c66a-47eb-a956-4f9346951de7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:20.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f067114-c66a-47eb-a956-4f9346951de7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:25.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90c74bc6-56c1-4afb-beef-e2e573636c50 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:25.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90c74bc6-56c1-4afb-beef-e2e573636c50 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@460eceb2 2021-05-04T00:43:25.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90c74bc6-56c1-4afb-beef-e2e573636c50 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:25.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90c74bc6-56c1-4afb-beef-e2e573636c50 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:26.454Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9486d9e-4532-4349-87a4-c07ba11171f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:26.454Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9486d9e-4532-4349-87a4-c07ba11171f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a3c8525 2021-05-04T00:43:26.454Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9486d9e-4532-4349-87a4-c07ba11171f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:26.454Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9486d9e-4532-4349-87a4-c07ba11171f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d22e4efd-7ea4-42c8-95c0-6cecc7e22fd2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d22e4efd-7ea4-42c8-95c0-6cecc7e22fd2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@179855ee 2021-05-04T00:43:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d22e4efd-7ea4-42c8-95c0-6cecc7e22fd2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d22e4efd-7ea4-42c8-95c0-6cecc7e22fd2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:32.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf529746-85ed-4e6d-b01d-5c96a22dcb2a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:32.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf529746-85ed-4e6d-b01d-5c96a22dcb2a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c266ddc 2021-05-04T00:43:32.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf529746-85ed-4e6d-b01d-5c96a22dcb2a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:32.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf529746-85ed-4e6d-b01d-5c96a22dcb2a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:43:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:43:36.455Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1358670-bc06-403f-a2e6-60afdb6bcef1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:36.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1358670-bc06-403f-a2e6-60afdb6bcef1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c6d5bb4 2021-05-04T00:43:36.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1358670-bc06-403f-a2e6-60afdb6bcef1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:36.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1358670-bc06-403f-a2e6-60afdb6bcef1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:43:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:43:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:43:38.475Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54b8aab6-e983-4384-a1b5-e55b59960b6f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:38.475Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54b8aab6-e983-4384-a1b5-e55b59960b6f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3266f6b1 2021-05-04T00:43:38.475Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54b8aab6-e983-4384-a1b5-e55b59960b6f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:38.475Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54b8aab6-e983-4384-a1b5-e55b59960b6f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3c5884b-e07f-4856-ae13-889707edeec1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3c5884b-e07f-4856-ae13-889707edeec1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6784f236 2021-05-04T00:43:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3c5884b-e07f-4856-ae13-889707edeec1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3c5884b-e07f-4856-ae13-889707edeec1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:44.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c0b2efa-7696-48db-8cad-34f51f850f06 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:44.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c0b2efa-7696-48db-8cad-34f51f850f06 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48633b97 2021-05-04T00:43:44.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c0b2efa-7696-48db-8cad-34f51f850f06 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:44.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c0b2efa-7696-48db-8cad-34f51f850f06 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:48.180Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6a55531-9791-4c4b-ae7a-703183c0c2ad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:48.180Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6a55531-9791-4c4b-ae7a-703183c0c2ad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e1569d4 2021-05-04T00:43:48.180Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6a55531-9791-4c4b-ae7a-703183c0c2ad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:48.180Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6a55531-9791-4c4b-ae7a-703183c0c2ad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:50.173Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:43:50.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:43:50.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=954c9fec-6e9d-40f9-b697-44ef60e441b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:50.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=954c9fec-6e9d-40f9-b697-44ef60e441b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b43c5d9 2021-05-04T00:43:50.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=954c9fec-6e9d-40f9-b697-44ef60e441b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:50.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=954c9fec-6e9d-40f9-b697-44ef60e441b9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:52.054Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:43:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba27999d-0a26-4d63-8500-eb072f9c8844 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba27999d-0a26-4d63-8500-eb072f9c8844 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b1683c5 2021-05-04T00:43:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba27999d-0a26-4d63-8500-eb072f9c8844 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba27999d-0a26-4d63-8500-eb072f9c8844 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:56.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c22e1af-2c54-465d-a7b9-aa4cefe8661c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:56.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c22e1af-2c54-465d-a7b9-aa4cefe8661c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a78a6b3 2021-05-04T00:43:56.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c22e1af-2c54-465d-a7b9-aa4cefe8661c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:56.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c22e1af-2c54-465d-a7b9-aa4cefe8661c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:43:59.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8eed9038-4d76-4d29-b28b-c760cebc5dfe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:43:59.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8eed9038-4d76-4d29-b28b-c760cebc5dfe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f8637e1 2021-05-04T00:43:59.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8eed9038-4d76-4d29-b28b-c760cebc5dfe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:43:59.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8eed9038-4d76-4d29-b28b-c760cebc5dfe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:03.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c88c85a4-6c62-45a0-a6cb-5aaedb8fe1e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:03.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c88c85a4-6c62-45a0-a6cb-5aaedb8fe1e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a7e40c5 2021-05-04T00:44:03.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c88c85a4-6c62-45a0-a6cb-5aaedb8fe1e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:03.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c88c85a4-6c62-45a0-a6cb-5aaedb8fe1e6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:44:05.184Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:44:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=009b789f-ae79-4b74-8939-8fabc46c4ad8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=009b789f-ae79-4b74-8939-8fabc46c4ad8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5509e6e4 2021-05-04T00:44:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=009b789f-ae79-4b74-8939-8fabc46c4ad8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=009b789f-ae79-4b74-8939-8fabc46c4ad8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:08.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7906a04-8cdd-4208-bf40-b709d50aead0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:08.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7906a04-8cdd-4208-bf40-b709d50aead0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a9b9fc8 2021-05-04T00:44:08.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7906a04-8cdd-4208-bf40-b709d50aead0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:08.851Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7906a04-8cdd-4208-bf40-b709d50aead0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:11.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ecc63e-bbd5-4ec7-a720-b74e3b2747b3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:11.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ecc63e-bbd5-4ec7-a720-b74e3b2747b3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42c8845 2021-05-04T00:44:11.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ecc63e-bbd5-4ec7-a720-b74e3b2747b3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:11.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ecc63e-bbd5-4ec7-a720-b74e3b2747b3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:14.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3852fe5-d234-4b34-8914-c19f0d7f2425 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:14.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3852fe5-d234-4b34-8914-c19f0d7f2425 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c8c986d 2021-05-04T00:44:14.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3852fe5-d234-4b34-8914-c19f0d7f2425 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:14.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3852fe5-d234-4b34-8914-c19f0d7f2425 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:17.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad587224-9e6b-46d9-8f2a-396d89644b98 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:17.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad587224-9e6b-46d9-8f2a-396d89644b98 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46a87bde 2021-05-04T00:44:17.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad587224-9e6b-46d9-8f2a-396d89644b98 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:17.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad587224-9e6b-46d9-8f2a-396d89644b98 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:44:20.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:44:20.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a938c9f6-88a6-483e-8ea0-ddddbeca78b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:20.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a938c9f6-88a6-483e-8ea0-ddddbeca78b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@109bd91c 2021-05-04T00:44:20.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a938c9f6-88a6-483e-8ea0-ddddbeca78b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:20.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a938c9f6-88a6-483e-8ea0-ddddbeca78b4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:23.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb460027-d59e-49c4-ad6c-7da1f114e6bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:23.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb460027-d59e-49c4-ad6c-7da1f114e6bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@709505ea 2021-05-04T00:44:23.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb460027-d59e-49c4-ad6c-7da1f114e6bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:23.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb460027-d59e-49c4-ad6c-7da1f114e6bd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:25.846Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=874e4cd6-893f-4f63-9fc4-da987aab7b6c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:25.846Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=874e4cd6-893f-4f63-9fc4-da987aab7b6c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4862a465 2021-05-04T00:44:25.847Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=874e4cd6-893f-4f63-9fc4-da987aab7b6c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:25.847Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=874e4cd6-893f-4f63-9fc4-da987aab7b6c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:29.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=034cf1fa-d15b-42d4-9dad-3e847f5e34d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:29.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=034cf1fa-d15b-42d4-9dad-3e847f5e34d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d91503c 2021-05-04T00:44:29.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=034cf1fa-d15b-42d4-9dad-3e847f5e34d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:29.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=034cf1fa-d15b-42d4-9dad-3e847f5e34d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:31.991Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f157047c-6f84-476a-b250-5f68fe5e7e21 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:31.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f157047c-6f84-476a-b250-5f68fe5e7e21 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38c23ab 2021-05-04T00:44:31.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f157047c-6f84-476a-b250-5f68fe5e7e21 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:31.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f157047c-6f84-476a-b250-5f68fe5e7e21 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:44:35.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:44:35.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f6537d8-6feb-4681-a464-85617767ce9c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:35.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f6537d8-6feb-4681-a464-85617767ce9c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b6edc65 2021-05-04T00:44:35.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f6537d8-6feb-4681-a464-85617767ce9c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:35.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f6537d8-6feb-4681-a464-85617767ce9c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:44:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:44:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:44:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8403ba3-bad2-4b53-9c1b-d2091ef61117 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8403ba3-bad2-4b53-9c1b-d2091ef61117 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6720ae1 2021-05-04T00:44:38.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8403ba3-bad2-4b53-9c1b-d2091ef61117 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:38.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8403ba3-bad2-4b53-9c1b-d2091ef61117 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4d157e-405a-4727-9b90-6490f88ced08 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4d157e-405a-4727-9b90-6490f88ced08 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4125baa3 2021-05-04T00:44:41.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4d157e-405a-4727-9b90-6490f88ced08 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:41.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4d157e-405a-4727-9b90-6490f88ced08 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:44.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89d87a92-9993-4000-a280-2caccc0c7ddf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:44.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89d87a92-9993-4000-a280-2caccc0c7ddf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23564857 2021-05-04T00:44:44.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89d87a92-9993-4000-a280-2caccc0c7ddf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:44.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89d87a92-9993-4000-a280-2caccc0c7ddf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:47.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e29ffd88-99c1-4047-a350-81d442c6c8fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:47.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e29ffd88-99c1-4047-a350-81d442c6c8fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e7635ac 2021-05-04T00:44:47.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e29ffd88-99c1-4047-a350-81d442c6c8fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:47.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e29ffd88-99c1-4047-a350-81d442c6c8fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:44:50.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:44:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bae0e877-42a8-43fe-a4c3-3381f97d11b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bae0e877-42a8-43fe-a4c3-3381f97d11b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18450381 2021-05-04T00:44:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bae0e877-42a8-43fe-a4c3-3381f97d11b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bae0e877-42a8-43fe-a4c3-3381f97d11b8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:52.104Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:44:53.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67318518-4c89-4dd8-be77-3ecd02bee960 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:53.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67318518-4c89-4dd8-be77-3ecd02bee960 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74594288 2021-05-04T00:44:53.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67318518-4c89-4dd8-be77-3ecd02bee960 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:53.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67318518-4c89-4dd8-be77-3ecd02bee960 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:57.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:44:57.207Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:44:59.052Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86b9ec0d-e3f2-41df-b3c4-b3e556c01d05 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:44:59.052Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86b9ec0d-e3f2-41df-b3c4-b3e556c01d05 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65dca8a4 2021-05-04T00:44:59.052Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86b9ec0d-e3f2-41df-b3c4-b3e556c01d05 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:44:59.052Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86b9ec0d-e3f2-41df-b3c4-b3e556c01d05 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:44:59.303Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:45:02.059Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=86b9ec0d-e3f2-41df-b3c4-b3e556c01d05 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:45:02.060Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=86b9ec0d-e3f2-41df-b3c4-b3e556c01d05 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:45:02.060Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=86b9ec0d-e3f2-41df-b3c4-b3e556c01d05 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:45:02.060Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=86b9ec0d-e3f2-41df-b3c4-b3e556c01d05 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:45:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:45:05.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:45:06.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be4d9bf4-4291-48cb-855f-17079f085433 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:06.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be4d9bf4-4291-48cb-855f-17079f085433 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a383ae6 2021-05-04T00:45:06.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be4d9bf4-4291-48cb-855f-17079f085433 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:06.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be4d9bf4-4291-48cb-855f-17079f085433 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:06.605Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=be4d9bf4-4291-48cb-855f-17079f085433 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:45:06.606Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=be4d9bf4-4291-48cb-855f-17079f085433 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:45:07.855Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2c87f87-a6f1-4ae0-a6b3-a6f158e98a91 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:07.855Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2c87f87-a6f1-4ae0-a6b3-a6f158e98a91 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@542231fb 2021-05-04T00:45:07.855Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2c87f87-a6f1-4ae0-a6b3-a6f158e98a91 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:07.855Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2c87f87-a6f1-4ae0-a6b3-a6f158e98a91 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:09.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da437150-3c95-4ef0-bf63-77ab0d5b0521 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:09.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da437150-3c95-4ef0-bf63-77ab0d5b0521 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54bb3a53 2021-05-04T00:45:09.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da437150-3c95-4ef0-bf63-77ab0d5b0521 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:09.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da437150-3c95-4ef0-bf63-77ab0d5b0521 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:10.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49a2a1fd-e800-40c8-bec2-12d03679ba94 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:10.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49a2a1fd-e800-40c8-bec2-12d03679ba94 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f4fc528 2021-05-04T00:45:10.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49a2a1fd-e800-40c8-bec2-12d03679ba94 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:10.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49a2a1fd-e800-40c8-bec2-12d03679ba94 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:11.750Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7fd9af5-aaba-4be7-8891-7c25f5711717 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:11.750Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7fd9af5-aaba-4be7-8891-7c25f5711717 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ee6bfc1 2021-05-04T00:45:11.750Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7fd9af5-aaba-4be7-8891-7c25f5711717 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:11.751Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7fd9af5-aaba-4be7-8891-7c25f5711717 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8a59b1c-7b5a-4e9d-a446-bfdd8f6a4cd9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8a59b1c-7b5a-4e9d-a446-bfdd8f6a4cd9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b8efe0f 2021-05-04T00:45:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8a59b1c-7b5a-4e9d-a446-bfdd8f6a4cd9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8a59b1c-7b5a-4e9d-a446-bfdd8f6a4cd9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ab0ada3-0292-481c-825c-94788f7e5beb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ab0ada3-0292-481c-825c-94788f7e5beb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66ea288f 2021-05-04T00:45:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ab0ada3-0292-481c-825c-94788f7e5beb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ab0ada3-0292-481c-825c-94788f7e5beb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:45:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:45:20.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f9b4d64-665e-4b21-9f38-4a44f90f06ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:20.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f9b4d64-665e-4b21-9f38-4a44f90f06ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@573dd7ff 2021-05-04T00:45:20.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f9b4d64-665e-4b21-9f38-4a44f90f06ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:20.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f9b4d64-665e-4b21-9f38-4a44f90f06ea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:23.043Z [qtp580718781-17] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=2140cbe5-184e-4a8e-99da-cc8d56e41e53 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:45:23.042226Z InvocationID=1eb7aa7e-a006-4e47-8bc5-518d827bcd19 LogTimestamp=2021-05-04T00:45:23.042379Z Entering 2021-05-04T00:45:23.043Z [qtp580718781-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=2140cbe5-184e-4a8e-99da-cc8d56e41e53 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:45:23.042226Z InvocationID=1eb7aa7e-a006-4e47-8bc5-518d827bcd19 LogTimestamp=2021-05-04T00:45:23.042379Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:45:23.047Z [qtp580718781-17] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=4 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:45:23.043Z InvocationID=8238b8d1-c24f-4182-b91f-fceabd8981a3 LogTimestamp=2021-05-04T00:45:23.047181Z ResponseDesc=OK Exiting. 2021-05-04T00:45:23.047Z [qtp580718781-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:45:23.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f336a6a1-13c6-4ab2-890b-d0a5568b1755 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:23.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f336a6a1-13c6-4ab2-890b-d0a5568b1755 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c3e726e 2021-05-04T00:45:23.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f336a6a1-13c6-4ab2-890b-d0a5568b1755 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:23.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f336a6a1-13c6-4ab2-890b-d0a5568b1755 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:24.094Z [qtp580718781-16] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=8839ebbb-983c-409a-a44a-8efd8e3fad4c InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:45:24.094253Z InvocationID=862d41be-3389-4c57-b5fa-66fff912718f LogTimestamp=2021-05-04T00:45:24.094314Z Entering 2021-05-04T00:45:24.094Z [qtp580718781-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=8839ebbb-983c-409a-a44a-8efd8e3fad4c InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:45:24.094253Z InvocationID=862d41be-3389-4c57-b5fa-66fff912718f LogTimestamp=2021-05-04T00:45:24.094314Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:45:24.101Z [qtp580718781-16] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:45:24.094Z InvocationID=20a03b3f-860d-44a1-9fc4-bd9d87894303 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/Otc70hukCdMWWdfV will send credentials over a clear channel. 2021-05-04T00:45:24.101Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:45:24.094Z InvocationID=20a03b3f-860d-44a1-9fc4-bd9d87894303 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/Otc70hukCdMWWdfV (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:45:24.142Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:45:24.094Z InvocationID=20a03b3f-860d-44a1-9fc4-bd9d87894303 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:45:24.174Z [qtp580718781-16] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "Otc70hukCdMWWdfV" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:45:24.094Z InvocationID=20a03b3f-860d-44a1-9fc4-bd9d87894303 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/Otc70hukCdMWWdfV will send credentials over a clear channel. 2021-05-04T00:45:24.175Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "Otc70hukCdMWWdfV" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:45:24.094Z InvocationID=20a03b3f-860d-44a1-9fc4-bd9d87894303 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/Otc70hukCdMWWdfV (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:45:24.243Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "Otc70hukCdMWWdfV" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:45:24.094Z InvocationID=20a03b3f-860d-44a1-9fc4-bd9d87894303 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:45:24.260Z [qtp580718781-16] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "ves-openapi-manager" API_KEY = "Otc70hukCdMWWdfV" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=166 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:45:24.094Z InvocationID=20a03b3f-860d-44a1-9fc4-bd9d87894303 LogTimestamp=2021-05-04T00:45:24.260034Z ResponseDesc=OK Exiting. 2021-05-04T00:45:24.260Z [qtp580718781-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:45:26.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcd16ff4-4d89-445e-89f4-da66eb22854f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:26.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcd16ff4-4d89-445e-89f4-da66eb22854f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50dc8360 2021-05-04T00:45:26.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcd16ff4-4d89-445e-89f4-da66eb22854f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:26.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcd16ff4-4d89-445e-89f4-da66eb22854f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:29.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=befecbe8-e6d9-4ad3-a498-057f5ad3eb78 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:29.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=befecbe8-e6d9-4ad3-a498-057f5ad3eb78 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67c567a4 2021-05-04T00:45:29.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=befecbe8-e6d9-4ad3-a498-057f5ad3eb78 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:29.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=befecbe8-e6d9-4ad3-a498-057f5ad3eb78 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:32.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96736cc-cadd-4376-af04-d5712889eec5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:32.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96736cc-cadd-4376-af04-d5712889eec5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5efec68c 2021-05-04T00:45:32.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96736cc-cadd-4376-af04-d5712889eec5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:32.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96736cc-cadd-4376-af04-d5712889eec5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:45:35.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:45:35.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02c6cd55-75cb-4830-b78c-b379d5e9a1fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:35.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02c6cd55-75cb-4830-b78c-b379d5e9a1fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61ec723f 2021-05-04T00:45:35.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02c6cd55-75cb-4830-b78c-b379d5e9a1fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:35.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02c6cd55-75cb-4830-b78c-b379d5e9a1fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:45:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:45:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:45:38.281Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea6662dd-729d-4660-a8f5-9d043f32250b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:38.281Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea6662dd-729d-4660-a8f5-9d043f32250b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@551f3ae1 2021-05-04T00:45:38.281Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea6662dd-729d-4660-a8f5-9d043f32250b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:38.281Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea6662dd-729d-4660-a8f5-9d043f32250b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:41.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4bfdbb9-b2cd-4789-8f4d-c253780752d3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:41.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4bfdbb9-b2cd-4789-8f4d-c253780752d3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19ab8fa1 2021-05-04T00:45:41.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4bfdbb9-b2cd-4789-8f4d-c253780752d3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:41.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4bfdbb9-b2cd-4789-8f4d-c253780752d3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e96f6823-b5df-4eff-8abd-655b68c2d7f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e96f6823-b5df-4eff-8abd-655b68c2d7f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3149e0be 2021-05-04T00:45:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e96f6823-b5df-4eff-8abd-655b68c2d7f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e96f6823-b5df-4eff-8abd-655b68c2d7f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:47.127Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52390b5d-f724-48dd-a7b8-fab0ef678b3f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:47.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52390b5d-f724-48dd-a7b8-fab0ef678b3f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33361122 2021-05-04T00:45:47.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52390b5d-f724-48dd-a7b8-fab0ef678b3f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:47.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52390b5d-f724-48dd-a7b8-fab0ef678b3f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:45:50.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:45:50.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ef74dc9-b949-4400-a767-18a70b491f4d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:50.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ef74dc9-b949-4400-a767-18a70b491f4d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f52ed97 2021-05-04T00:45:50.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ef74dc9-b949-4400-a767-18a70b491f4d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:50.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ef74dc9-b949-4400-a767-18a70b491f4d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:52.043Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:45:53.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e25f7f4-4b27-4b18-9a88-c33fdb24107d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:53.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e25f7f4-4b27-4b18-9a88-c33fdb24107d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ac1af5b 2021-05-04T00:45:53.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e25f7f4-4b27-4b18-9a88-c33fdb24107d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:53.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e25f7f4-4b27-4b18-9a88-c33fdb24107d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:56.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74c28efd-cb6e-40cf-8634-528bf490890d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:56.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74c28efd-cb6e-40cf-8634-528bf490890d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24f3e8ad 2021-05-04T00:45:56.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74c28efd-cb6e-40cf-8634-528bf490890d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:56.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74c28efd-cb6e-40cf-8634-528bf490890d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:45:59.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6a9ea40-86c0-4e29-a096-4c14bfdc9999 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:45:59.252Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6a9ea40-86c0-4e29-a096-4c14bfdc9999 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27cf4210 2021-05-04T00:45:59.252Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6a9ea40-86c0-4e29-a096-4c14bfdc9999 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:45:59.252Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6a9ea40-86c0-4e29-a096-4c14bfdc9999 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:02.256Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c6a9ea40-86c0-4e29-a096-4c14bfdc9999 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:46:02.257Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c6a9ea40-86c0-4e29-a096-4c14bfdc9999 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:46:02.257Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c6a9ea40-86c0-4e29-a096-4c14bfdc9999 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:46:02.257Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c6a9ea40-86c0-4e29-a096-4c14bfdc9999 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:46:03.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05b1a05d-b8dc-4da9-916a-c50e53734e7e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:03.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05b1a05d-b8dc-4da9-916a-c50e53734e7e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ff92d2b 2021-05-04T00:46:03.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05b1a05d-b8dc-4da9-916a-c50e53734e7e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:03.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05b1a05d-b8dc-4da9-916a-c50e53734e7e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:04.013Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=05b1a05d-b8dc-4da9-916a-c50e53734e7e ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:46:04.014Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=05b1a05d-b8dc-4da9-916a-c50e53734e7e ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:46:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:46:05.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:46:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=084f9925-91de-441c-bb28-01166b84756f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:05.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=084f9925-91de-441c-bb28-01166b84756f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4318cdc7 2021-05-04T00:46:05.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=084f9925-91de-441c-bb28-01166b84756f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:05.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=084f9925-91de-441c-bb28-01166b84756f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:08.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40b623a0-4b52-41d3-aa52-fed3a0037fe6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:08.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40b623a0-4b52-41d3-aa52-fed3a0037fe6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e9ac876 2021-05-04T00:46:08.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40b623a0-4b52-41d3-aa52-fed3a0037fe6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:08.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40b623a0-4b52-41d3-aa52-fed3a0037fe6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:11.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a8af170-7f71-4349-953b-1f0487c523dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:11.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a8af170-7f71-4349-953b-1f0487c523dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78ec3efb 2021-05-04T00:46:11.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a8af170-7f71-4349-953b-1f0487c523dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:11.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a8af170-7f71-4349-953b-1f0487c523dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:14.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f7b5ab7-83ab-4c0f-8b64-e6a7bd8d510f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f7b5ab7-83ab-4c0f-8b64-e6a7bd8d510f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f09a73a 2021-05-04T00:46:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f7b5ab7-83ab-4c0f-8b64-e6a7bd8d510f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f7b5ab7-83ab-4c0f-8b64-e6a7bd8d510f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:17.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d35b43e-2a9f-4d92-bf0b-c2519d5bfd7f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:17.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d35b43e-2a9f-4d92-bf0b-c2519d5bfd7f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a7a21e9 2021-05-04T00:46:17.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d35b43e-2a9f-4d92-bf0b-c2519d5bfd7f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:17.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d35b43e-2a9f-4d92-bf0b-c2519d5bfd7f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:46:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:46:20.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c64675a-fb70-41eb-9d9b-3b137e7bfbb2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:20.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c64675a-fb70-41eb-9d9b-3b137e7bfbb2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62eb85a8 2021-05-04T00:46:20.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c64675a-fb70-41eb-9d9b-3b137e7bfbb2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:20.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c64675a-fb70-41eb-9d9b-3b137e7bfbb2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:23.609Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f751383a-c367-4330-841d-f6415ab3e3d0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:23.609Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f751383a-c367-4330-841d-f6415ab3e3d0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7be974e8 2021-05-04T00:46:23.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f751383a-c367-4330-841d-f6415ab3e3d0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:23.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f751383a-c367-4330-841d-f6415ab3e3d0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:26.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b2a1893-4056-4dba-a9d9-79eaac5da064 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:26.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b2a1893-4056-4dba-a9d9-79eaac5da064 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19bb3cd8 2021-05-04T00:46:26.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b2a1893-4056-4dba-a9d9-79eaac5da064 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:26.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b2a1893-4056-4dba-a9d9-79eaac5da064 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f82b052e-0feb-4962-a3c6-51a3557a04cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f82b052e-0feb-4962-a3c6-51a3557a04cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6235236f 2021-05-04T00:46:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f82b052e-0feb-4962-a3c6-51a3557a04cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f82b052e-0feb-4962-a3c6-51a3557a04cf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:32.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17e9b39d-140c-4b88-a218-3dc55b432875 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:32.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17e9b39d-140c-4b88-a218-3dc55b432875 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ccd3c44 2021-05-04T00:46:32.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17e9b39d-140c-4b88-a218-3dc55b432875 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:32.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17e9b39d-140c-4b88-a218-3dc55b432875 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:46:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:46:35.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2dce7b11-5a7f-4851-9971-644b468a8a1e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:35.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2dce7b11-5a7f-4851-9971-644b468a8a1e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6cb35d01 2021-05-04T00:46:35.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2dce7b11-5a7f-4851-9971-644b468a8a1e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:35.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2dce7b11-5a7f-4851-9971-644b468a8a1e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:46:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:46:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:46:38.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fad5685-7b60-4154-b887-02cc4650d697 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:38.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fad5685-7b60-4154-b887-02cc4650d697 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c2ee1b3 2021-05-04T00:46:38.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fad5685-7b60-4154-b887-02cc4650d697 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:38.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fad5685-7b60-4154-b887-02cc4650d697 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:41.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d7deac5-9208-4ad5-9e45-147720beb74c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:41.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d7deac5-9208-4ad5-9e45-147720beb74c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@674b9102 2021-05-04T00:46:41.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d7deac5-9208-4ad5-9e45-147720beb74c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:41.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d7deac5-9208-4ad5-9e45-147720beb74c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:44.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973ed03e-605d-4731-8327-e97820117a40 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:44.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973ed03e-605d-4731-8327-e97820117a40 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52d1aeb3 2021-05-04T00:46:44.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973ed03e-605d-4731-8327-e97820117a40 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:44.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973ed03e-605d-4731-8327-e97820117a40 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:46.878Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=965c136f-b6ed-4e9b-8542-e676f51a0ebf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:46.878Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=965c136f-b6ed-4e9b-8542-e676f51a0ebf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24fcbabd 2021-05-04T00:46:46.878Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=965c136f-b6ed-4e9b-8542-e676f51a0ebf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:46.878Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=965c136f-b6ed-4e9b-8542-e676f51a0ebf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:46:50.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:46:50.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f633c43-818a-4700-a110-d8c266cc41ed PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:50.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f633c43-818a-4700-a110-d8c266cc41ed PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f84f4e8 2021-05-04T00:46:50.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f633c43-818a-4700-a110-d8c266cc41ed PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:50.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f633c43-818a-4700-a110-d8c266cc41ed PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:52.059Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:46:53.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33160bd5-3c63-46fc-9888-b0e7d0749c38 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:53.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33160bd5-3c63-46fc-9888-b0e7d0749c38 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@447ca99b 2021-05-04T00:46:53.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33160bd5-3c63-46fc-9888-b0e7d0749c38 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:53.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33160bd5-3c63-46fc-9888-b0e7d0749c38 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:56.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ca0dd40-992f-49b7-bb9d-42a84f2b2aee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:56.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ca0dd40-992f-49b7-bb9d-42a84f2b2aee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b872391 2021-05-04T00:46:56.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ca0dd40-992f-49b7-bb9d-42a84f2b2aee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:56.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ca0dd40-992f-49b7-bb9d-42a84f2b2aee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:46:59.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90cb5593-672f-4bdc-938e-855bd4bfd89a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:46:59.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90cb5593-672f-4bdc-938e-855bd4bfd89a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5aef4b0e 2021-05-04T00:46:59.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90cb5593-672f-4bdc-938e-855bd4bfd89a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:46:59.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90cb5593-672f-4bdc-938e-855bd4bfd89a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:02.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95a8adf5-b14e-45b9-93ec-671f5f90d734 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:02.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95a8adf5-b14e-45b9-93ec-671f5f90d734 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@214c70f 2021-05-04T00:47:02.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95a8adf5-b14e-45b9-93ec-671f5f90d734 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:02.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95a8adf5-b14e-45b9-93ec-671f5f90d734 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:05.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:47:05.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:47:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a90351e5-f69c-4fff-bb86-f49aeeb7403d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a90351e5-f69c-4fff-bb86-f49aeeb7403d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@230586a7 2021-05-04T00:47:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a90351e5-f69c-4fff-bb86-f49aeeb7403d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:05.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a90351e5-f69c-4fff-bb86-f49aeeb7403d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:08.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bdb2402-d066-4d4e-b36a-b2934eba8685 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:08.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bdb2402-d066-4d4e-b36a-b2934eba8685 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13cb2fd 2021-05-04T00:47:08.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bdb2402-d066-4d4e-b36a-b2934eba8685 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:08.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bdb2402-d066-4d4e-b36a-b2934eba8685 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:11.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2add0840-0609-47c5-8e9a-c18a5a518c88 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:11.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2add0840-0609-47c5-8e9a-c18a5a518c88 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@212b2eef 2021-05-04T00:47:11.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2add0840-0609-47c5-8e9a-c18a5a518c88 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:11.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2add0840-0609-47c5-8e9a-c18a5a518c88 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:14.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbe54375-4225-4fc0-9626-09b0038cee3c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:14.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbe54375-4225-4fc0-9626-09b0038cee3c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a3b97a 2021-05-04T00:47:14.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbe54375-4225-4fc0-9626-09b0038cee3c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:14.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbe54375-4225-4fc0-9626-09b0038cee3c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:17.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73c1a6e3-070d-49bb-8091-a85aecd752c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:17.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73c1a6e3-070d-49bb-8091-a85aecd752c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a9c27d7 2021-05-04T00:47:17.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73c1a6e3-070d-49bb-8091-a85aecd752c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:17.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73c1a6e3-070d-49bb-8091-a85aecd752c1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:47:20.184Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:47:20.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbca6eb9-dcb2-4dfd-b5a6-91c9de31a140 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:20.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbca6eb9-dcb2-4dfd-b5a6-91c9de31a140 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e53c723 2021-05-04T00:47:20.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbca6eb9-dcb2-4dfd-b5a6-91c9de31a140 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:20.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbca6eb9-dcb2-4dfd-b5a6-91c9de31a140 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:23.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3560244-2f3f-497d-abc7-a8653fb23558 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:23.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3560244-2f3f-497d-abc7-a8653fb23558 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1677d050 2021-05-04T00:47:23.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3560244-2f3f-497d-abc7-a8653fb23558 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:23.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3560244-2f3f-497d-abc7-a8653fb23558 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:26.244Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e30028ce-90eb-4342-b1ae-dbbdcf5ddb0e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:26.244Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e30028ce-90eb-4342-b1ae-dbbdcf5ddb0e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e87c38d 2021-05-04T00:47:26.244Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e30028ce-90eb-4342-b1ae-dbbdcf5ddb0e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:26.245Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e30028ce-90eb-4342-b1ae-dbbdcf5ddb0e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:29.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a947ae0e-6450-4d62-a08a-349088dca71a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:29.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a947ae0e-6450-4d62-a08a-349088dca71a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@654eff15 2021-05-04T00:47:29.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a947ae0e-6450-4d62-a08a-349088dca71a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:29.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a947ae0e-6450-4d62-a08a-349088dca71a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:32.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2d17235-919a-475b-8efb-fda26de632f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:32.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2d17235-919a-475b-8efb-fda26de632f8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15544640 2021-05-04T00:47:32.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2d17235-919a-475b-8efb-fda26de632f8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:32.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2d17235-919a-475b-8efb-fda26de632f8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:47:35.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32bc7f0e-4829-472b-9714-c49e1477ca33 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:35.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32bc7f0e-4829-472b-9714-c49e1477ca33 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d89e1ea 2021-05-04T00:47:35.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32bc7f0e-4829-472b-9714-c49e1477ca33 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:35.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32bc7f0e-4829-472b-9714-c49e1477ca33 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:35.508Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:47:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:47:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:47:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:47:38.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bca45772-6f3f-4d76-875a-17305b7c40f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:38.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bca45772-6f3f-4d76-875a-17305b7c40f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d22d67d 2021-05-04T00:47:38.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bca45772-6f3f-4d76-875a-17305b7c40f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:38.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bca45772-6f3f-4d76-875a-17305b7c40f6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:41.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a1bfdd-8b02-45f7-ad02-e61c87fb71c0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:41.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a1bfdd-8b02-45f7-ad02-e61c87fb71c0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f86369b 2021-05-04T00:47:41.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a1bfdd-8b02-45f7-ad02-e61c87fb71c0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:41.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a1bfdd-8b02-45f7-ad02-e61c87fb71c0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b814c655-7122-4574-9370-8ea37aa3c601 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b814c655-7122-4574-9370-8ea37aa3c601 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2857156d 2021-05-04T00:47:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b814c655-7122-4574-9370-8ea37aa3c601 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b814c655-7122-4574-9370-8ea37aa3c601 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:47.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad3cdb51-f3cf-4bf4-9756-f72fa63a56db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:47.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad3cdb51-f3cf-4bf4-9756-f72fa63a56db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5aa3373c 2021-05-04T00:47:47.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad3cdb51-f3cf-4bf4-9756-f72fa63a56db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:47.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad3cdb51-f3cf-4bf4-9756-f72fa63a56db PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:47:50.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=602d6649-2778-44e4-bab4-92c826dd9326 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:50.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=602d6649-2778-44e4-bab4-92c826dd9326 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4129832b 2021-05-04T00:47:50.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=602d6649-2778-44e4-bab4-92c826dd9326 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:50.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=602d6649-2778-44e4-bab4-92c826dd9326 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:53.288Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:47:53.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=460ed913-8e69-435c-9086-d7158ac83303 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:53.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=460ed913-8e69-435c-9086-d7158ac83303 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f35477b 2021-05-04T00:47:53.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=460ed913-8e69-435c-9086-d7158ac83303 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:53.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=460ed913-8e69-435c-9086-d7158ac83303 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:56.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a11bd7ab-fed7-45dd-964f-af3c396d527e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:56.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a11bd7ab-fed7-45dd-964f-af3c396d527e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fd9db8d 2021-05-04T00:47:56.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a11bd7ab-fed7-45dd-964f-af3c396d527e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:56.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a11bd7ab-fed7-45dd-964f-af3c396d527e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:58.171Z [UEB-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend probably lost connectivity to UEB Cluster. Please check the logs for more information. 2021-05-04T00:47:59.098Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3014810-51fc-4bd8-ad51-b3cb5af798cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:47:59.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3014810-51fc-4bd8-ad51-b3cb5af798cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c00099d 2021-05-04T00:47:59.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3014810-51fc-4bd8-ad51-b3cb5af798cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:47:59.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3014810-51fc-4bd8-ad51-b3cb5af798cb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:47:59.130Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e3014810-51fc-4bd8-ad51-b3cb5af798cb ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:47:59.131Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e3014810-51fc-4bd8-ad51-b3cb5af798cb ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:48:02.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c7f82f2-b7eb-4d51-88f4-dfb8c020fd90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:02.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c7f82f2-b7eb-4d51-88f4-dfb8c020fd90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31bde5ed 2021-05-04T00:48:02.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c7f82f2-b7eb-4d51-88f4-dfb8c020fd90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:02.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c7f82f2-b7eb-4d51-88f4-dfb8c020fd90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:05.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb892ca6-4176-4ccd-9f57-01804f090af0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:05.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb892ca6-4176-4ccd-9f57-01804f090af0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74f6623e 2021-05-04T00:48:05.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb892ca6-4176-4ccd-9f57-01804f090af0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:05.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb892ca6-4176-4ccd-9f57-01804f090af0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:08.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9572707f-2f85-4223-8562-b06a1dbdfd07 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:08.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9572707f-2f85-4223-8562-b06a1dbdfd07 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45c2ca9e 2021-05-04T00:48:08.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9572707f-2f85-4223-8562-b06a1dbdfd07 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:08.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9572707f-2f85-4223-8562-b06a1dbdfd07 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:11.150Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=414f1fbc-bbf8-4a89-b1cf-0846e78d4bc4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:11.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=414f1fbc-bbf8-4a89-b1cf-0846e78d4bc4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79b27f5a 2021-05-04T00:48:11.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=414f1fbc-bbf8-4a89-b1cf-0846e78d4bc4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:11.151Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=414f1fbc-bbf8-4a89-b1cf-0846e78d4bc4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78de73ff-8ffb-4174-b2b6-31cf89da913a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78de73ff-8ffb-4174-b2b6-31cf89da913a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@277e7f0c 2021-05-04T00:48:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78de73ff-8ffb-4174-b2b6-31cf89da913a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78de73ff-8ffb-4174-b2b6-31cf89da913a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:17.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce256acd-9d76-4019-8318-a94845d3efab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:17.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce256acd-9d76-4019-8318-a94845d3efab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60de1e4f 2021-05-04T00:48:17.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce256acd-9d76-4019-8318-a94845d3efab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:17.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce256acd-9d76-4019-8318-a94845d3efab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:20.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e0bb875-44c3-4a1a-8962-d1986e6e6813 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:20.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e0bb875-44c3-4a1a-8962-d1986e6e6813 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@339baef0 2021-05-04T00:48:20.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e0bb875-44c3-4a1a-8962-d1986e6e6813 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:20.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e0bb875-44c3-4a1a-8962-d1986e6e6813 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ad086db-a7e7-4394-a9a9-253b40e18e51 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ad086db-a7e7-4394-a9a9-253b40e18e51 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59b94f76 2021-05-04T00:48:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ad086db-a7e7-4394-a9a9-253b40e18e51 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ad086db-a7e7-4394-a9a9-253b40e18e51 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:26.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9a534fb-5423-4582-ac48-b2d13f1ca371 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:26.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9a534fb-5423-4582-ac48-b2d13f1ca371 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d6bdd52 2021-05-04T00:48:26.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9a534fb-5423-4582-ac48-b2d13f1ca371 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:26.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9a534fb-5423-4582-ac48-b2d13f1ca371 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:29.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0294098d-cd88-438a-8349-69cfc6a10ef2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:29.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0294098d-cd88-438a-8349-69cfc6a10ef2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d5796d6 2021-05-04T00:48:29.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0294098d-cd88-438a-8349-69cfc6a10ef2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:29.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0294098d-cd88-438a-8349-69cfc6a10ef2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:32.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1566bab6-d3a8-4a4c-9a17-a77c894aa5dd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:32.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1566bab6-d3a8-4a4c-9a17-a77c894aa5dd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@735557a2 2021-05-04T00:48:32.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1566bab6-d3a8-4a4c-9a17-a77c894aa5dd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:32.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1566bab6-d3a8-4a4c-9a17-a77c894aa5dd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e25ecb3f-bd69-4ea5-b62a-c896f538ed21 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e25ecb3f-bd69-4ea5-b62a-c896f538ed21 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60047441 2021-05-04T00:48:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e25ecb3f-bd69-4ea5-b62a-c896f538ed21 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e25ecb3f-bd69-4ea5-b62a-c896f538ed21 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:48:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:48:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:48:38.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05816d8c-d5db-4afb-8683-60ff12231c98 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:38.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05816d8c-d5db-4afb-8683-60ff12231c98 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33d6b45e 2021-05-04T00:48:38.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05816d8c-d5db-4afb-8683-60ff12231c98 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:38.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05816d8c-d5db-4afb-8683-60ff12231c98 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:41.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b508b18f-00ba-4607-b728-b91a6f1c516b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:41.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b508b18f-00ba-4607-b728-b91a6f1c516b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@129675d4 2021-05-04T00:48:41.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b508b18f-00ba-4607-b728-b91a6f1c516b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:41.112Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b508b18f-00ba-4607-b728-b91a6f1c516b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:44.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=538ca4ce-346f-47fd-9812-39eb696bcbe5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:44.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=538ca4ce-346f-47fd-9812-39eb696bcbe5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b27c863 2021-05-04T00:48:44.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=538ca4ce-346f-47fd-9812-39eb696bcbe5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:44.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=538ca4ce-346f-47fd-9812-39eb696bcbe5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:47.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fd32009-f83d-4d7e-991f-3a16b2d67bb9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:47.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fd32009-f83d-4d7e-991f-3a16b2d67bb9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8c20b3a 2021-05-04T00:48:47.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fd32009-f83d-4d7e-991f-3a16b2d67bb9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:47.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fd32009-f83d-4d7e-991f-3a16b2d67bb9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:50.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c912305-68fe-43ac-a47f-2edf97595a6c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:50.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c912305-68fe-43ac-a47f-2edf97595a6c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e302a38 2021-05-04T00:48:50.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c912305-68fe-43ac-a47f-2edf97595a6c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:50.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c912305-68fe-43ac-a47f-2edf97595a6c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:53.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbc8771b-58c3-44e4-b2af-4bad6c0262ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:53.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbc8771b-58c3-44e4-b2af-4bad6c0262ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f29c6ff 2021-05-04T00:48:53.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbc8771b-58c3-44e4-b2af-4bad6c0262ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:53.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbc8771b-58c3-44e4-b2af-4bad6c0262ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:56.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e60a577-12c8-403b-840a-6605bd9f59a4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:56.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e60a577-12c8-403b-840a-6605bd9f59a4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ab877a9 2021-05-04T00:48:56.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e60a577-12c8-403b-840a-6605bd9f59a4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:56.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e60a577-12c8-403b-840a-6605bd9f59a4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:48:59.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5043e8f2-ca4a-4a3c-9b3c-7f0516ca7b73 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:48:59.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5043e8f2-ca4a-4a3c-9b3c-7f0516ca7b73 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42bb2c1f 2021-05-04T00:48:59.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5043e8f2-ca4a-4a3c-9b3c-7f0516ca7b73 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:48:59.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5043e8f2-ca4a-4a3c-9b3c-7f0516ca7b73 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:02.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=047c3967-ef99-420e-bd05-74cd8024ed14 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:02.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=047c3967-ef99-420e-bd05-74cd8024ed14 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d55af9a 2021-05-04T00:49:02.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=047c3967-ef99-420e-bd05-74cd8024ed14 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:02.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=047c3967-ef99-420e-bd05-74cd8024ed14 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:05.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f2baeed-f6ad-408f-994c-57a09064cc79 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:05.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f2baeed-f6ad-408f-994c-57a09064cc79 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c2a96f1 2021-05-04T00:49:05.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f2baeed-f6ad-408f-994c-57a09064cc79 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:05.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f2baeed-f6ad-408f-994c-57a09064cc79 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:08.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5081e1c-a795-4be4-a318-94e57ce5ae93 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:08.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5081e1c-a795-4be4-a318-94e57ce5ae93 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3536b9e2 2021-05-04T00:49:08.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5081e1c-a795-4be4-a318-94e57ce5ae93 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:08.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5081e1c-a795-4be4-a318-94e57ce5ae93 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:11.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bd14bbe-3fa7-4945-9f02-c2c7d6de06c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:11.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bd14bbe-3fa7-4945-9f02-c2c7d6de06c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e76a08f 2021-05-04T00:49:11.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bd14bbe-3fa7-4945-9f02-c2c7d6de06c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:11.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bd14bbe-3fa7-4945-9f02-c2c7d6de06c6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=236c2801-3129-4896-be73-08a4acdb9838 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=236c2801-3129-4896-be73-08a4acdb9838 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e3ff3a4 2021-05-04T00:49:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=236c2801-3129-4896-be73-08a4acdb9838 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=236c2801-3129-4896-be73-08a4acdb9838 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:17.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f6b88ea-ef94-4280-836e-8ed8658be8a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:17.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f6b88ea-ef94-4280-836e-8ed8658be8a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@365a798c 2021-05-04T00:49:17.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f6b88ea-ef94-4280-836e-8ed8658be8a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:17.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f6b88ea-ef94-4280-836e-8ed8658be8a2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c66dc089-4ba3-4e2f-b407-91c7c96c68ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c66dc089-4ba3-4e2f-b407-91c7c96c68ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@36a33f0c 2021-05-04T00:49:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c66dc089-4ba3-4e2f-b407-91c7c96c68ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:20.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c66dc089-4ba3-4e2f-b407-91c7c96c68ea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:23.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e7a2823-d4a9-4714-bc62-1f75764d81c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:23.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e7a2823-d4a9-4714-bc62-1f75764d81c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d4a838f 2021-05-04T00:49:23.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e7a2823-d4a9-4714-bc62-1f75764d81c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:23.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e7a2823-d4a9-4714-bc62-1f75764d81c2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:26.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53fb8035-d0f8-46f3-bad4-49e5e1ca0962 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:26.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53fb8035-d0f8-46f3-bad4-49e5e1ca0962 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c59f751 2021-05-04T00:49:26.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53fb8035-d0f8-46f3-bad4-49e5e1ca0962 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:26.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53fb8035-d0f8-46f3-bad4-49e5e1ca0962 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:29.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82f65b1e-49a7-4883-81ac-b52e399761ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:29.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82f65b1e-49a7-4883-81ac-b52e399761ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79aa6b4f 2021-05-04T00:49:29.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82f65b1e-49a7-4883-81ac-b52e399761ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:29.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82f65b1e-49a7-4883-81ac-b52e399761ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24e21c88-c9aa-4e03-b038-681f4b80a974 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24e21c88-c9aa-4e03-b038-681f4b80a974 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@565e7029 2021-05-04T00:49:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24e21c88-c9aa-4e03-b038-681f4b80a974 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24e21c88-c9aa-4e03-b038-681f4b80a974 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:35.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf7af454-f9b0-4bfe-90a6-1fda3c634624 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:35.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf7af454-f9b0-4bfe-90a6-1fda3c634624 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f71d796 2021-05-04T00:49:35.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf7af454-f9b0-4bfe-90a6-1fda3c634624 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:35.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf7af454-f9b0-4bfe-90a6-1fda3c634624 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:38.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6d002fb-9924-40f1-97fa-9de9bce9ad30 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:38.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6d002fb-9924-40f1-97fa-9de9bce9ad30 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cc14e56 2021-05-04T00:49:38.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6d002fb-9924-40f1-97fa-9de9bce9ad30 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:38.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6d002fb-9924-40f1-97fa-9de9bce9ad30 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:41.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e261e01-0f1e-4a09-b69c-236f2a1d5bd7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:41.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e261e01-0f1e-4a09-b69c-236f2a1d5bd7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@569ba292 2021-05-04T00:49:41.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e261e01-0f1e-4a09-b69c-236f2a1d5bd7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:41.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e261e01-0f1e-4a09-b69c-236f2a1d5bd7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:44.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ef467a1-8b09-48d2-9e0f-dc21744a2991 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:44.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ef467a1-8b09-48d2-9e0f-dc21744a2991 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73e15f50 2021-05-04T00:49:44.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ef467a1-8b09-48d2-9e0f-dc21744a2991 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:44.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ef467a1-8b09-48d2-9e0f-dc21744a2991 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:47.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52410387-2257-4b69-814e-27c0c64b7c9e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:47.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52410387-2257-4b69-814e-27c0c64b7c9e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@528dfbdd 2021-05-04T00:49:47.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52410387-2257-4b69-814e-27c0c64b7c9e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:47.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52410387-2257-4b69-814e-27c0c64b7c9e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98ab6b94-c405-4650-9a9f-c01db23569bf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98ab6b94-c405-4650-9a9f-c01db23569bf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2448ac24 2021-05-04T00:49:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98ab6b94-c405-4650-9a9f-c01db23569bf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98ab6b94-c405-4650-9a9f-c01db23569bf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:53.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0306d17-566f-41b5-bbe5-fc2c5a02cdc3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:53.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0306d17-566f-41b5-bbe5-fc2c5a02cdc3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2fcc050d 2021-05-04T00:49:53.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0306d17-566f-41b5-bbe5-fc2c5a02cdc3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:53.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0306d17-566f-41b5-bbe5-fc2c5a02cdc3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:56.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dde0d8a-bcb9-4342-b140-2528291ca6d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:56.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dde0d8a-bcb9-4342-b140-2528291ca6d8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56aed47b 2021-05-04T00:49:56.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dde0d8a-bcb9-4342-b140-2528291ca6d8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:56.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dde0d8a-bcb9-4342-b140-2528291ca6d8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:49:59.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=601c5ddc-7c9f-4233-bc23-b35144aadeb7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:49:59.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=601c5ddc-7c9f-4233-bc23-b35144aadeb7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b408e60 2021-05-04T00:49:59.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=601c5ddc-7c9f-4233-bc23-b35144aadeb7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:49:59.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=601c5ddc-7c9f-4233-bc23-b35144aadeb7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:02.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f77f58be-db99-46a3-851e-479946844c6e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:02.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f77f58be-db99-46a3-851e-479946844c6e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a8826a0 2021-05-04T00:50:02.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f77f58be-db99-46a3-851e-479946844c6e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:02.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f77f58be-db99-46a3-851e-479946844c6e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f73639f-294e-48c1-8685-3b5092dcf7d4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f73639f-294e-48c1-8685-3b5092dcf7d4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45a3be91 2021-05-04T00:50:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f73639f-294e-48c1-8685-3b5092dcf7d4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:05.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f73639f-294e-48c1-8685-3b5092dcf7d4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:08.069Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01i3quq9kn59n71l7u7wyh091tr267 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0tl6o7le801sw1xph8196tzotv257 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0y50rld11nubibsrz4wspwspr260 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0164kyrhvrae161017q3e13xtmn263 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node02hrsv0psmazeykbv9adm1wxs266 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01dnz6duahvix0bkuwqm6dspos261 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0l1nu7njbr2mu1e2k70y00b9sp262 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0108umhigkgl2o12kiktrai0mo258 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01adptmra7gjds1qf2x05s9ua13259 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01gj4kgvs3iu10or4ta16lxte0264 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0661q1xcgl5p5u7fpl0t6txt1265 2021-05-04T00:50:08.070Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0lro5675z4ggx1mveacj7r25z3268 2021-05-04T00:50:08.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=730b4a1f-f325-427f-a02d-4d4f4bfaf7cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:08.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=730b4a1f-f325-427f-a02d-4d4f4bfaf7cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@650eb752 2021-05-04T00:50:08.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=730b4a1f-f325-427f-a02d-4d4f4bfaf7cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:08.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=730b4a1f-f325-427f-a02d-4d4f4bfaf7cd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:11.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61409d59-efbb-411c-8310-29c1aa6a52c3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:11.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61409d59-efbb-411c-8310-29c1aa6a52c3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a52161d 2021-05-04T00:50:11.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61409d59-efbb-411c-8310-29c1aa6a52c3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:11.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61409d59-efbb-411c-8310-29c1aa6a52c3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac8b63c-59ca-4f97-bed9-f99a968ecd20 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:14.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac8b63c-59ca-4f97-bed9-f99a968ecd20 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41249cd1 2021-05-04T00:50:14.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac8b63c-59ca-4f97-bed9-f99a968ecd20 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:14.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac8b63c-59ca-4f97-bed9-f99a968ecd20 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:17.125Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ea881e0-a988-4551-8998-40f1ae2c620c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:17.125Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ea881e0-a988-4551-8998-40f1ae2c620c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@192bf552 2021-05-04T00:50:17.125Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ea881e0-a988-4551-8998-40f1ae2c620c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:17.125Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ea881e0-a988-4551-8998-40f1ae2c620c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:20.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b344a943-fed1-4497-9e34-e6b148db828d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:20.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b344a943-fed1-4497-9e34-e6b148db828d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@90f6502 2021-05-04T00:50:20.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b344a943-fed1-4497-9e34-e6b148db828d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:20.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b344a943-fed1-4497-9e34-e6b148db828d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:23.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06c16e70-781a-4ecb-8d22-8188fbe19814 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:23.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06c16e70-781a-4ecb-8d22-8188fbe19814 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13f768c0 2021-05-04T00:50:23.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06c16e70-781a-4ecb-8d22-8188fbe19814 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:23.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06c16e70-781a-4ecb-8d22-8188fbe19814 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:26.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6a677a2-fb51-45e8-857d-31bb3749fcde PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:26.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6a677a2-fb51-45e8-857d-31bb3749fcde PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7650ac7b 2021-05-04T00:50:26.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6a677a2-fb51-45e8-857d-31bb3749fcde PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:26.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6a677a2-fb51-45e8-857d-31bb3749fcde PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:29.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73a9a8fd-6386-4222-878f-073751a32251 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:29.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73a9a8fd-6386-4222-878f-073751a32251 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30975686 2021-05-04T00:50:29.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73a9a8fd-6386-4222-878f-073751a32251 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:29.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73a9a8fd-6386-4222-878f-073751a32251 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:32.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c706f08-4de1-4441-a005-0b9f4335e5c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:32.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c706f08-4de1-4441-a005-0b9f4335e5c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@424212e3 2021-05-04T00:50:32.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c706f08-4de1-4441-a005-0b9f4335e5c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:32.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c706f08-4de1-4441-a005-0b9f4335e5c6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e18ebc2a-2300-4ea0-8a56-fbd1f88b90b2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e18ebc2a-2300-4ea0-8a56-fbd1f88b90b2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6adbc303 2021-05-04T00:50:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e18ebc2a-2300-4ea0-8a56-fbd1f88b90b2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e18ebc2a-2300-4ea0-8a56-fbd1f88b90b2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c5204b1-5c9d-453a-850c-93ed38f84a5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c5204b1-5c9d-453a-850c-93ed38f84a5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@199b8957 2021-05-04T00:50:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c5204b1-5c9d-453a-850c-93ed38f84a5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c5204b1-5c9d-453a-850c-93ed38f84a5f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:41.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7929508-355f-4454-a64c-a5273e8f0d26 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:41.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7929508-355f-4454-a64c-a5273e8f0d26 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13b2cf84 2021-05-04T00:50:41.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7929508-355f-4454-a64c-a5273e8f0d26 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:41.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7929508-355f-4454-a64c-a5273e8f0d26 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b71d2d13-ef55-44eb-bd08-086b944d5f90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b71d2d13-ef55-44eb-bd08-086b944d5f90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4135121b 2021-05-04T00:50:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b71d2d13-ef55-44eb-bd08-086b944d5f90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b71d2d13-ef55-44eb-bd08-086b944d5f90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:47.012Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c977034-012d-445b-8837-2e1a1b2e29e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:47.012Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c977034-012d-445b-8837-2e1a1b2e29e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@754202e5 2021-05-04T00:50:47.012Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c977034-012d-445b-8837-2e1a1b2e29e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:47.012Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c977034-012d-445b-8837-2e1a1b2e29e9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=088261cd-6ed8-410b-a6f0-df96e1491a7d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=088261cd-6ed8-410b-a6f0-df96e1491a7d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3056fc2b 2021-05-04T00:50:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=088261cd-6ed8-410b-a6f0-df96e1491a7d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:50.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=088261cd-6ed8-410b-a6f0-df96e1491a7d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:53.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ca5611f-56e9-49e9-a098-634196d2fd2a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:53.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ca5611f-56e9-49e9-a098-634196d2fd2a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@351ddb4a 2021-05-04T00:50:53.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ca5611f-56e9-49e9-a098-634196d2fd2a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:53.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ca5611f-56e9-49e9-a098-634196d2fd2a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:56.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6cf0fca8-52dc-4372-bb8e-a8f833d35a6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:56.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6cf0fca8-52dc-4372-bb8e-a8f833d35a6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c51a238 2021-05-04T00:50:56.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6cf0fca8-52dc-4372-bb8e-a8f833d35a6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:56.326Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6cf0fca8-52dc-4372-bb8e-a8f833d35a6a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:50:59.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c7461b7-828c-47b1-aadf-f65344b46aa2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:50:59.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c7461b7-828c-47b1-aadf-f65344b46aa2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39e1d67d 2021-05-04T00:50:59.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c7461b7-828c-47b1-aadf-f65344b46aa2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:50:59.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c7461b7-828c-47b1-aadf-f65344b46aa2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:02.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a75dc861-8053-4ba0-a4ad-6a473ba5e023 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:02.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a75dc861-8053-4ba0-a4ad-6a473ba5e023 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ed1d2ce 2021-05-04T00:51:02.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a75dc861-8053-4ba0-a4ad-6a473ba5e023 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:02.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a75dc861-8053-4ba0-a4ad-6a473ba5e023 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:05.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=908c3b8d-03f0-46d9-b76c-984fbefe22d4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:05.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=908c3b8d-03f0-46d9-b76c-984fbefe22d4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@654d99d5 2021-05-04T00:51:05.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=908c3b8d-03f0-46d9-b76c-984fbefe22d4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:05.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=908c3b8d-03f0-46d9-b76c-984fbefe22d4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:08.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72c647d4-414b-47e3-8360-5e7f4395d39f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:08.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72c647d4-414b-47e3-8360-5e7f4395d39f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34dcb726 2021-05-04T00:51:08.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72c647d4-414b-47e3-8360-5e7f4395d39f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:08.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72c647d4-414b-47e3-8360-5e7f4395d39f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:11.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d035ff55-438a-4583-9cc9-a9d22b6fbda6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:11.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d035ff55-438a-4583-9cc9-a9d22b6fbda6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@339cdfe 2021-05-04T00:51:11.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d035ff55-438a-4583-9cc9-a9d22b6fbda6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:11.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d035ff55-438a-4583-9cc9-a9d22b6fbda6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:14.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=965bb6de-c5f8-4c49-a01e-6969509e5ec1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:14.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=965bb6de-c5f8-4c49-a01e-6969509e5ec1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33ce28f5 2021-05-04T00:51:14.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=965bb6de-c5f8-4c49-a01e-6969509e5ec1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:14.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=965bb6de-c5f8-4c49-a01e-6969509e5ec1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:17.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aeeaf0d5-e31d-479c-b12e-3eee8585adaf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:17.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aeeaf0d5-e31d-479c-b12e-3eee8585adaf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a448c68 2021-05-04T00:51:17.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aeeaf0d5-e31d-479c-b12e-3eee8585adaf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:17.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aeeaf0d5-e31d-479c-b12e-3eee8585adaf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:20.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f62fa319-4c01-4ecc-92eb-95702bfd1970 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:20.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f62fa319-4c01-4ecc-92eb-95702bfd1970 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b707981 2021-05-04T00:51:20.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f62fa319-4c01-4ecc-92eb-95702bfd1970 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:20.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f62fa319-4c01-4ecc-92eb-95702bfd1970 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:23.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4d98976-e063-4270-9429-b88d5ac3784f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:23.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4d98976-e063-4270-9429-b88d5ac3784f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26eb0f2c 2021-05-04T00:51:23.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4d98976-e063-4270-9429-b88d5ac3784f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4d98976-e063-4270-9429-b88d5ac3784f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:26.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e8776e1-a5ef-42ad-b058-57f264705e75 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:26.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e8776e1-a5ef-42ad-b058-57f264705e75 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64a00284 2021-05-04T00:51:26.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e8776e1-a5ef-42ad-b058-57f264705e75 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:26.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e8776e1-a5ef-42ad-b058-57f264705e75 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:29.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c1ee03a-cce6-42c3-aa9e-78e438d9518c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:29.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c1ee03a-cce6-42c3-aa9e-78e438d9518c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a19f5b4 2021-05-04T00:51:29.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c1ee03a-cce6-42c3-aa9e-78e438d9518c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:29.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c1ee03a-cce6-42c3-aa9e-78e438d9518c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:32.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1d75bcc-c661-49db-94e8-36b7fdd29829 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:32.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1d75bcc-c661-49db-94e8-36b7fdd29829 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12cd3707 2021-05-04T00:51:32.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1d75bcc-c661-49db-94e8-36b7fdd29829 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:32.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1d75bcc-c661-49db-94e8-36b7fdd29829 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:35.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=333ac38b-b878-4002-ade2-e31c439b0110 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:35.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=333ac38b-b878-4002-ade2-e31c439b0110 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c21c81d 2021-05-04T00:51:35.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=333ac38b-b878-4002-ade2-e31c439b0110 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:35.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=333ac38b-b878-4002-ade2-e31c439b0110 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:38.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2cb4745-ecf7-43be-b3cf-4af5c3d646a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:38.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2cb4745-ecf7-43be-b3cf-4af5c3d646a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@871533f 2021-05-04T00:51:38.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2cb4745-ecf7-43be-b3cf-4af5c3d646a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:38.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2cb4745-ecf7-43be-b3cf-4af5c3d646a6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:41.107Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b907c709-9578-4294-aede-49cfe5194155 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:41.107Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b907c709-9578-4294-aede-49cfe5194155 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d70a3d0 2021-05-04T00:51:41.107Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b907c709-9578-4294-aede-49cfe5194155 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:41.107Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b907c709-9578-4294-aede-49cfe5194155 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:44.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97d1b861-dc81-4f26-89e1-a1cbc07fdcf8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:44.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97d1b861-dc81-4f26-89e1-a1cbc07fdcf8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@467ede76 2021-05-04T00:51:44.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97d1b861-dc81-4f26-89e1-a1cbc07fdcf8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:44.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97d1b861-dc81-4f26-89e1-a1cbc07fdcf8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:47.249Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e783139f-e9b7-400e-ad75-fd7c3329f1dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:47.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e783139f-e9b7-400e-ad75-fd7c3329f1dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@247d9784 2021-05-04T00:51:47.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e783139f-e9b7-400e-ad75-fd7c3329f1dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:47.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e783139f-e9b7-400e-ad75-fd7c3329f1dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:49.927Z [qtp580718781-17] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=891a2a22-b632-4ca7-8347-aa315f6cf34d InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:51:49.927388Z InvocationID=59685708-fdde-4290-926b-c43c1f295fcb LogTimestamp=2021-05-04T00:51:49.927527Z Entering 2021-05-04T00:51:49.928Z [qtp580718781-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=891a2a22-b632-4ca7-8347-aa315f6cf34d InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:51:49.927388Z InvocationID=59685708-fdde-4290-926b-c43c1f295fcb LogTimestamp=2021-05-04T00:51:49.927527Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:51:49.931Z [qtp580718781-17] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=3 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:51:49.928Z InvocationID=d5a2d296-0268-4478-b7be-00d104a4b1ec LogTimestamp=2021-05-04T00:51:49.931349Z ResponseDesc=OK Exiting. 2021-05-04T00:51:49.932Z [qtp580718781-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:51:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c04cde7b-24bd-4564-855c-0460b3d37cf5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c04cde7b-24bd-4564-855c-0460b3d37cf5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2697d09f 2021-05-04T00:51:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c04cde7b-24bd-4564-855c-0460b3d37cf5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c04cde7b-24bd-4564-855c-0460b3d37cf5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7c9143c-9e22-48e3-a5a3-a06bab8caf9c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7c9143c-9e22-48e3-a5a3-a06bab8caf9c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e4199be 2021-05-04T00:51:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7c9143c-9e22-48e3-a5a3-a06bab8caf9c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7c9143c-9e22-48e3-a5a3-a06bab8caf9c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:56.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61abe5df-d540-4055-b0d6-613b7c419b31 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:56.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61abe5df-d540-4055-b0d6-613b7c419b31 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e7c696e 2021-05-04T00:51:56.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61abe5df-d540-4055-b0d6-613b7c419b31 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:56.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61abe5df-d540-4055-b0d6-613b7c419b31 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:51:59.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1e73db7-6a45-4653-a74d-19a9cfde7051 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:51:59.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1e73db7-6a45-4653-a74d-19a9cfde7051 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66751a92 2021-05-04T00:51:59.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1e73db7-6a45-4653-a74d-19a9cfde7051 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:51:59.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1e73db7-6a45-4653-a74d-19a9cfde7051 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:02.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=221821a9-0bd6-4d48-accc-eea170955071 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:02.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=221821a9-0bd6-4d48-accc-eea170955071 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7efe3deb 2021-05-04T00:52:02.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=221821a9-0bd6-4d48-accc-eea170955071 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:02.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=221821a9-0bd6-4d48-accc-eea170955071 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:05.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e17f83b7-e6ef-4085-842f-509ec6431265 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:05.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e17f83b7-e6ef-4085-842f-509ec6431265 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ce0d7e3 2021-05-04T00:52:05.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e17f83b7-e6ef-4085-842f-509ec6431265 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:05.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e17f83b7-e6ef-4085-842f-509ec6431265 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:08.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4850ccf-f822-404d-ad4b-bc7f48b5b94e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:08.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4850ccf-f822-404d-ad4b-bc7f48b5b94e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@211ae408 2021-05-04T00:52:08.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4850ccf-f822-404d-ad4b-bc7f48b5b94e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:08.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4850ccf-f822-404d-ad4b-bc7f48b5b94e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=912a301b-d89f-4f0b-ad04-970f1cb51192 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=912a301b-d89f-4f0b-ad04-970f1cb51192 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ff5cb33 2021-05-04T00:52:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=912a301b-d89f-4f0b-ad04-970f1cb51192 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:11.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=912a301b-d89f-4f0b-ad04-970f1cb51192 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:14.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e0cd6ad-3558-4cce-9a45-aa00d553d022 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:14.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e0cd6ad-3558-4cce-9a45-aa00d553d022 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@606bb243 2021-05-04T00:52:14.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e0cd6ad-3558-4cce-9a45-aa00d553d022 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:14.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e0cd6ad-3558-4cce-9a45-aa00d553d022 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:17.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879f24ae-e0e3-45c6-b174-266a13f8029f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:17.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879f24ae-e0e3-45c6-b174-266a13f8029f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ade3e62 2021-05-04T00:52:17.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879f24ae-e0e3-45c6-b174-266a13f8029f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:17.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879f24ae-e0e3-45c6-b174-266a13f8029f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:20.429Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be3b51ae-c2b4-48ef-8233-83ffacdfbb54 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:20.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be3b51ae-c2b4-48ef-8233-83ffacdfbb54 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c12a195 2021-05-04T00:52:20.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be3b51ae-c2b4-48ef-8233-83ffacdfbb54 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:20.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be3b51ae-c2b4-48ef-8233-83ffacdfbb54 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:23.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7dee8fc7-9aa9-49c1-a0ad-bd1b3f6bf1e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:23.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7dee8fc7-9aa9-49c1-a0ad-bd1b3f6bf1e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d7f4177 2021-05-04T00:52:23.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7dee8fc7-9aa9-49c1-a0ad-bd1b3f6bf1e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:23.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7dee8fc7-9aa9-49c1-a0ad-bd1b3f6bf1e9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:26.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05a36022-2a5a-45ea-bf01-8d659a186bbc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:26.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05a36022-2a5a-45ea-bf01-8d659a186bbc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b1531 2021-05-04T00:52:26.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05a36022-2a5a-45ea-bf01-8d659a186bbc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:26.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05a36022-2a5a-45ea-bf01-8d659a186bbc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:29.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4a43afb-af58-4b14-a8d4-02898782660a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:29.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4a43afb-af58-4b14-a8d4-02898782660a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7bed067a 2021-05-04T00:52:29.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4a43afb-af58-4b14-a8d4-02898782660a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:29.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4a43afb-af58-4b14-a8d4-02898782660a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:31.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20ab3aa9-be78-41e3-827b-3bed2fc2a4b7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:31.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20ab3aa9-be78-41e3-827b-3bed2fc2a4b7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@129a1e9b 2021-05-04T00:52:31.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20ab3aa9-be78-41e3-827b-3bed2fc2a4b7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:31.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20ab3aa9-be78-41e3-827b-3bed2fc2a4b7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:35.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dab50e1f-d970-4c97-a6c6-0a1b341447ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:35.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dab50e1f-d970-4c97-a6c6-0a1b341447ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55299146 2021-05-04T00:52:35.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dab50e1f-d970-4c97-a6c6-0a1b341447ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:35.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dab50e1f-d970-4c97-a6c6-0a1b341447ca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:38.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f359aaa1-9c4d-43e1-a93a-cbcc2fad6846 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:38.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f359aaa1-9c4d-43e1-a93a-cbcc2fad6846 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@de3844f 2021-05-04T00:52:38.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f359aaa1-9c4d-43e1-a93a-cbcc2fad6846 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:38.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f359aaa1-9c4d-43e1-a93a-cbcc2fad6846 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:41.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=887cc921-2949-4a33-a908-1fe06c8566d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:41.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=887cc921-2949-4a33-a908-1fe06c8566d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e2e9d5f 2021-05-04T00:52:41.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=887cc921-2949-4a33-a908-1fe06c8566d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:41.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=887cc921-2949-4a33-a908-1fe06c8566d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:44.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78d6f922-9c44-4b90-806f-8a3c0fb8f5ad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:44.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78d6f922-9c44-4b90-806f-8a3c0fb8f5ad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c8226c7 2021-05-04T00:52:44.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78d6f922-9c44-4b90-806f-8a3c0fb8f5ad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:44.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78d6f922-9c44-4b90-806f-8a3c0fb8f5ad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:47.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d0867df-edd6-4934-95bc-e61037f859e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:47.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d0867df-edd6-4934-95bc-e61037f859e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45b82fb0 2021-05-04T00:52:47.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d0867df-edd6-4934-95bc-e61037f859e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:47.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d0867df-edd6-4934-95bc-e61037f859e9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:50.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd7c828-8e0c-4f08-89ae-24c25a34bbd1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:50.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd7c828-8e0c-4f08-89ae-24c25a34bbd1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54e80ab4 2021-05-04T00:52:50.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd7c828-8e0c-4f08-89ae-24c25a34bbd1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:50.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1dd7c828-8e0c-4f08-89ae-24c25a34bbd1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:53.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9385fd35-52db-4296-a9b9-e136ba7e492b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:53.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9385fd35-52db-4296-a9b9-e136ba7e492b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ec6a113 2021-05-04T00:52:53.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9385fd35-52db-4296-a9b9-e136ba7e492b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:53.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9385fd35-52db-4296-a9b9-e136ba7e492b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:56.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e8f72b8-06ab-4608-b2ff-2c97e10f768d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:56.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e8f72b8-06ab-4608-b2ff-2c97e10f768d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@733f1c12 2021-05-04T00:52:56.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e8f72b8-06ab-4608-b2ff-2c97e10f768d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:56.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e8f72b8-06ab-4608-b2ff-2c97e10f768d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:52:59.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=848c3982-c543-43e9-80b3-df1072d19b60 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:52:59.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=848c3982-c543-43e9-80b3-df1072d19b60 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77fe686a 2021-05-04T00:52:59.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=848c3982-c543-43e9-80b3-df1072d19b60 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:52:59.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=848c3982-c543-43e9-80b3-df1072d19b60 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:02.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d93c9c42-7f1f-4fa7-bbf2-c07d9bad8a89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:02.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d93c9c42-7f1f-4fa7-bbf2-c07d9bad8a89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6357618b 2021-05-04T00:53:02.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d93c9c42-7f1f-4fa7-bbf2-c07d9bad8a89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d93c9c42-7f1f-4fa7-bbf2-c07d9bad8a89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=918be0ed-74cb-45ce-b8d1-480e0ef49d95 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:05.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=918be0ed-74cb-45ce-b8d1-480e0ef49d95 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68128acb 2021-05-04T00:53:05.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=918be0ed-74cb-45ce-b8d1-480e0ef49d95 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:05.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=918be0ed-74cb-45ce-b8d1-480e0ef49d95 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:08.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee9503f0-94f7-4788-a1b3-f7be4b417893 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:08.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee9503f0-94f7-4788-a1b3-f7be4b417893 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63c6d457 2021-05-04T00:53:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee9503f0-94f7-4788-a1b3-f7be4b417893 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee9503f0-94f7-4788-a1b3-f7be4b417893 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:11.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f318274d-ef10-4ac0-96c0-fd4622d3c71d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:11.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f318274d-ef10-4ac0-96c0-fd4622d3c71d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@677f5d7d 2021-05-04T00:53:11.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f318274d-ef10-4ac0-96c0-fd4622d3c71d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:11.954Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f318274d-ef10-4ac0-96c0-fd4622d3c71d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:14.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40c431b2-f5c9-4e3a-9b25-2f21b80b2d65 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:14.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40c431b2-f5c9-4e3a-9b25-2f21b80b2d65 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f7c3320 2021-05-04T00:53:14.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40c431b2-f5c9-4e3a-9b25-2f21b80b2d65 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:14.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40c431b2-f5c9-4e3a-9b25-2f21b80b2d65 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:18.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70cff214-fa66-4a63-9964-4c25a2493d14 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:18.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70cff214-fa66-4a63-9964-4c25a2493d14 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43c4a2d6 2021-05-04T00:53:18.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70cff214-fa66-4a63-9964-4c25a2493d14 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:18.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70cff214-fa66-4a63-9964-4c25a2493d14 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:20.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e59ecf42-a9dd-41fc-aa35-647aee291f1b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:20.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e59ecf42-a9dd-41fc-aa35-647aee291f1b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16f5b295 2021-05-04T00:53:20.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e59ecf42-a9dd-41fc-aa35-647aee291f1b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:20.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e59ecf42-a9dd-41fc-aa35-647aee291f1b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ada1657e-e9b2-400b-844b-40d7621f2a34 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ada1657e-e9b2-400b-844b-40d7621f2a34 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bfc8183 2021-05-04T00:53:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ada1657e-e9b2-400b-844b-40d7621f2a34 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:23.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ada1657e-e9b2-400b-844b-40d7621f2a34 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:26.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab1441bd-7639-4740-9746-2f8046ecac63 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:26.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab1441bd-7639-4740-9746-2f8046ecac63 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55c756ef 2021-05-04T00:53:26.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab1441bd-7639-4740-9746-2f8046ecac63 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:26.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab1441bd-7639-4740-9746-2f8046ecac63 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:29.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38120bc2-4606-437b-a4df-922553c74b97 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:29.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38120bc2-4606-437b-a4df-922553c74b97 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33f38c93 2021-05-04T00:53:29.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38120bc2-4606-437b-a4df-922553c74b97 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:29.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38120bc2-4606-437b-a4df-922553c74b97 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:32.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9e74050-0f4e-4b6c-bf4e-d15acf5c3f6e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:32.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9e74050-0f4e-4b6c-bf4e-d15acf5c3f6e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8a9f162 2021-05-04T00:53:32.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9e74050-0f4e-4b6c-bf4e-d15acf5c3f6e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:32.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9e74050-0f4e-4b6c-bf4e-d15acf5c3f6e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6dbfbe27-2081-4115-88a0-212f69348258 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6dbfbe27-2081-4115-88a0-212f69348258 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67b6996d 2021-05-04T00:53:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6dbfbe27-2081-4115-88a0-212f69348258 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6dbfbe27-2081-4115-88a0-212f69348258 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:39.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=704bbf98-6cc1-4805-8491-10207fee6249 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:39.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=704bbf98-6cc1-4805-8491-10207fee6249 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@829db5a 2021-05-04T00:53:39.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=704bbf98-6cc1-4805-8491-10207fee6249 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:39.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=704bbf98-6cc1-4805-8491-10207fee6249 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:42.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca79d76-50fe-4bd2-a9a9-60c8df6ed414 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:42.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca79d76-50fe-4bd2-a9a9-60c8df6ed414 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@271159d8 2021-05-04T00:53:42.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca79d76-50fe-4bd2-a9a9-60c8df6ed414 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:42.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca79d76-50fe-4bd2-a9a9-60c8df6ed414 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:44.884Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=949bc37b-63df-4e1a-9496-c3c6287d8538 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:44.884Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=949bc37b-63df-4e1a-9496-c3c6287d8538 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60748eb9 2021-05-04T00:53:44.884Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=949bc37b-63df-4e1a-9496-c3c6287d8538 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:44.884Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=949bc37b-63df-4e1a-9496-c3c6287d8538 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:47.053Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46064abd-17b4-4549-bf07-0dce4a98ecf6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:47.053Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46064abd-17b4-4549-bf07-0dce4a98ecf6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@335a1e2b 2021-05-04T00:53:47.054Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46064abd-17b4-4549-bf07-0dce4a98ecf6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:47.054Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46064abd-17b4-4549-bf07-0dce4a98ecf6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:50.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09fe1ca7-0d47-4624-a4e4-34ecde340119 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:50.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09fe1ca7-0d47-4624-a4e4-34ecde340119 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75eff011 2021-05-04T00:53:50.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09fe1ca7-0d47-4624-a4e4-34ecde340119 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:50.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09fe1ca7-0d47-4624-a4e4-34ecde340119 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:53.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=102b5103-0ca7-42fc-b4e8-1d855e076920 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:53.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=102b5103-0ca7-42fc-b4e8-1d855e076920 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@581da8ed 2021-05-04T00:53:53.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=102b5103-0ca7-42fc-b4e8-1d855e076920 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:53.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=102b5103-0ca7-42fc-b4e8-1d855e076920 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:56.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18ffa26f-62c6-4858-9291-c8ec7af337a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:56.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18ffa26f-62c6-4858-9291-c8ec7af337a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c58f411 2021-05-04T00:53:56.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18ffa26f-62c6-4858-9291-c8ec7af337a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:56.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18ffa26f-62c6-4858-9291-c8ec7af337a0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:53:59.346Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1f3c934-3611-4cb9-b8b4-a1778257b8f2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:53:59.346Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1f3c934-3611-4cb9-b8b4-a1778257b8f2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f900af7 2021-05-04T00:53:59.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1f3c934-3611-4cb9-b8b4-a1778257b8f2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:53:59.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1f3c934-3611-4cb9-b8b4-a1778257b8f2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:02.427Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6b82b12-da8b-45ce-bbf1-2fb842beb8b0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:02.427Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6b82b12-da8b-45ce-bbf1-2fb842beb8b0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7575026c 2021-05-04T00:54:02.428Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6b82b12-da8b-45ce-bbf1-2fb842beb8b0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:02.428Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6b82b12-da8b-45ce-bbf1-2fb842beb8b0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:05.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=446b248b-d3c2-4f03-b899-c26b2ab9973f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:05.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=446b248b-d3c2-4f03-b899-c26b2ab9973f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@401a96ef 2021-05-04T00:54:05.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=446b248b-d3c2-4f03-b899-c26b2ab9973f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:05.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=446b248b-d3c2-4f03-b899-c26b2ab9973f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:08.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16bff26c-9374-476b-93df-bf0dc3750bd3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:08.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16bff26c-9374-476b-93df-bf0dc3750bd3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bb91a81 2021-05-04T00:54:08.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16bff26c-9374-476b-93df-bf0dc3750bd3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:08.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16bff26c-9374-476b-93df-bf0dc3750bd3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:11.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c0f50ef-dd9e-44f2-b049-d96da546e6a5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:11.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c0f50ef-dd9e-44f2-b049-d96da546e6a5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64016cb3 2021-05-04T00:54:11.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c0f50ef-dd9e-44f2-b049-d96da546e6a5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:11.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c0f50ef-dd9e-44f2-b049-d96da546e6a5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c6487e3-d13c-48a3-9849-8ccc68659232 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c6487e3-d13c-48a3-9849-8ccc68659232 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39ba643d 2021-05-04T00:54:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c6487e3-d13c-48a3-9849-8ccc68659232 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:14.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c6487e3-d13c-48a3-9849-8ccc68659232 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:17.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d47f3730-dbba-464b-8ef0-8ce556a5d4b0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:17.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d47f3730-dbba-464b-8ef0-8ce556a5d4b0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d83300b 2021-05-04T00:54:17.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d47f3730-dbba-464b-8ef0-8ce556a5d4b0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:17.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d47f3730-dbba-464b-8ef0-8ce556a5d4b0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:20.598Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5b8bffe-23b9-49f2-a9a4-31237c469210 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:20.598Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5b8bffe-23b9-49f2-a9a4-31237c469210 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e3d4a28 2021-05-04T00:54:20.598Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5b8bffe-23b9-49f2-a9a4-31237c469210 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:20.598Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5b8bffe-23b9-49f2-a9a4-31237c469210 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:23.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de69d91a-6797-4ea8-8c64-be57d09b2105 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:23.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de69d91a-6797-4ea8-8c64-be57d09b2105 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1136c5a8 2021-05-04T00:54:23.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de69d91a-6797-4ea8-8c64-be57d09b2105 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:23.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de69d91a-6797-4ea8-8c64-be57d09b2105 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:26.354Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84ab779a-9082-4f26-8881-f6e6b63ba44c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:26.354Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84ab779a-9082-4f26-8881-f6e6b63ba44c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7bc7c03f 2021-05-04T00:54:26.354Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84ab779a-9082-4f26-8881-f6e6b63ba44c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:26.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84ab779a-9082-4f26-8881-f6e6b63ba44c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:29.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4dfa5cd-c3b8-4d93-a690-8c9dee02291f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:29.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4dfa5cd-c3b8-4d93-a690-8c9dee02291f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ff0b424 2021-05-04T00:54:29.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4dfa5cd-c3b8-4d93-a690-8c9dee02291f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:29.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4dfa5cd-c3b8-4d93-a690-8c9dee02291f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:31.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=203135d8-e46a-4775-87bd-1e39e1841847 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:31.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=203135d8-e46a-4775-87bd-1e39e1841847 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b88cbf8 2021-05-04T00:54:31.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=203135d8-e46a-4775-87bd-1e39e1841847 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:31.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=203135d8-e46a-4775-87bd-1e39e1841847 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:35.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=479fb79e-6dc3-4773-b34f-4bbe716e6b81 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:35.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=479fb79e-6dc3-4773-b34f-4bbe716e6b81 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cb44ea4 2021-05-04T00:54:35.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=479fb79e-6dc3-4773-b34f-4bbe716e6b81 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:35.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=479fb79e-6dc3-4773-b34f-4bbe716e6b81 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:38.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=07202ee6-8377-40a3-ae1b-0996c6520e46 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:38.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=07202ee6-8377-40a3-ae1b-0996c6520e46 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39b7af35 2021-05-04T00:54:38.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=07202ee6-8377-40a3-ae1b-0996c6520e46 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:38.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=07202ee6-8377-40a3-ae1b-0996c6520e46 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:41.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=069800cb-e780-454a-9989-e2a98c5929d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:41.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=069800cb-e780-454a-9989-e2a98c5929d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73ac6db3 2021-05-04T00:54:41.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=069800cb-e780-454a-9989-e2a98c5929d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:41.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=069800cb-e780-454a-9989-e2a98c5929d7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:44.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=909a10c8-1ad0-4a6f-8d97-94a72295086b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:44.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=909a10c8-1ad0-4a6f-8d97-94a72295086b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ee6bf35 2021-05-04T00:54:44.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=909a10c8-1ad0-4a6f-8d97-94a72295086b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:44.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=909a10c8-1ad0-4a6f-8d97-94a72295086b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:46.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=910954f1-ff06-43eb-b789-ddbaef1d7a89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:46.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=910954f1-ff06-43eb-b789-ddbaef1d7a89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30790d23 2021-05-04T00:54:46.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=910954f1-ff06-43eb-b789-ddbaef1d7a89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:46.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=910954f1-ff06-43eb-b789-ddbaef1d7a89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c82efc5b-f039-445d-85a4-d56df4f5cdbf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c82efc5b-f039-445d-85a4-d56df4f5cdbf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3433d21e 2021-05-04T00:54:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c82efc5b-f039-445d-85a4-d56df4f5cdbf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:50.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c82efc5b-f039-445d-85a4-d56df4f5cdbf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:53.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1eee3de-5410-44ec-9f43-f7d6207c98cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:53.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1eee3de-5410-44ec-9f43-f7d6207c98cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24e2e0de 2021-05-04T00:54:53.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1eee3de-5410-44ec-9f43-f7d6207c98cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:53.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1eee3de-5410-44ec-9f43-f7d6207c98cf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:56.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3e9b6c-b1c6-4296-af83-4062757627ab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:56.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3e9b6c-b1c6-4296-af83-4062757627ab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f26ec60 2021-05-04T00:54:56.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3e9b6c-b1c6-4296-af83-4062757627ab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:56.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3e9b6c-b1c6-4296-af83-4062757627ab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:54:59.279Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23778913-b4f2-4bf4-9f24-8e2cc1c588f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:54:59.279Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23778913-b4f2-4bf4-9f24-8e2cc1c588f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74ef6b2 2021-05-04T00:54:59.279Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23778913-b4f2-4bf4-9f24-8e2cc1c588f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:54:59.279Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23778913-b4f2-4bf4-9f24-8e2cc1c588f6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:02.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be686395-fdb1-4ee9-be8f-dadb857e32f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:02.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be686395-fdb1-4ee9-be8f-dadb857e32f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d476767 2021-05-04T00:55:02.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be686395-fdb1-4ee9-be8f-dadb857e32f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:02.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be686395-fdb1-4ee9-be8f-dadb857e32f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d61c497-6919-47dd-852d-94114bc074cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:05.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d61c497-6919-47dd-852d-94114bc074cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e909601 2021-05-04T00:55:05.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d61c497-6919-47dd-852d-94114bc074cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:05.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d61c497-6919-47dd-852d-94114bc074cd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0369e8f6-6e7c-454d-b960-b0a260a08ed6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0369e8f6-6e7c-454d-b960-b0a260a08ed6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@561d3c5f 2021-05-04T00:55:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0369e8f6-6e7c-454d-b960-b0a260a08ed6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:08.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0369e8f6-6e7c-454d-b960-b0a260a08ed6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:11.546Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d42abb88-a090-4e7b-9d8b-cb2566d90543 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:11.546Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d42abb88-a090-4e7b-9d8b-cb2566d90543 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2616bb48 2021-05-04T00:55:11.546Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d42abb88-a090-4e7b-9d8b-cb2566d90543 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:11.546Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d42abb88-a090-4e7b-9d8b-cb2566d90543 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a6550b3-54de-430a-8f86-45c76af96a47 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a6550b3-54de-430a-8f86-45c76af96a47 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@307c9cb2 2021-05-04T00:55:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a6550b3-54de-430a-8f86-45c76af96a47 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a6550b3-54de-430a-8f86-45c76af96a47 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:17.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72a6a347-d9a0-4ea5-9452-b094c721c65e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:17.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72a6a347-d9a0-4ea5-9452-b094c721c65e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ac62422 2021-05-04T00:55:17.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72a6a347-d9a0-4ea5-9452-b094c721c65e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:17.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72a6a347-d9a0-4ea5-9452-b094c721c65e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:20.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3572886d-8afa-437e-9323-b70e255bb226 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:20.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3572886d-8afa-437e-9323-b70e255bb226 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15060ae1 2021-05-04T00:55:20.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3572886d-8afa-437e-9323-b70e255bb226 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:20.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3572886d-8afa-437e-9323-b70e255bb226 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21bd9872-c2a0-4eff-9ea3-aeeda644903d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21bd9872-c2a0-4eff-9ea3-aeeda644903d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6592ff2 2021-05-04T00:55:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21bd9872-c2a0-4eff-9ea3-aeeda644903d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:23.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21bd9872-c2a0-4eff-9ea3-aeeda644903d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:26.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b483591-eb5b-4b58-8e9a-3da51b2b1de1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:26.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b483591-eb5b-4b58-8e9a-3da51b2b1de1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5214ba87 2021-05-04T00:55:26.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b483591-eb5b-4b58-8e9a-3da51b2b1de1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:26.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b483591-eb5b-4b58-8e9a-3da51b2b1de1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:29.751Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0f5a9db-32da-4290-8305-e1ad4b9b07c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:29.751Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0f5a9db-32da-4290-8305-e1ad4b9b07c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@496873d3 2021-05-04T00:55:29.751Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0f5a9db-32da-4290-8305-e1ad4b9b07c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:29.751Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0f5a9db-32da-4290-8305-e1ad4b9b07c2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:32.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25d29dcd-799e-44fc-a306-b7d932ec1f08 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:32.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25d29dcd-799e-44fc-a306-b7d932ec1f08 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c9c291d 2021-05-04T00:55:32.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25d29dcd-799e-44fc-a306-b7d932ec1f08 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:32.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25d29dcd-799e-44fc-a306-b7d932ec1f08 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:35.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8eb8234-ea3a-4123-b437-cfe1787bff44 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:35.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8eb8234-ea3a-4123-b437-cfe1787bff44 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2375adc3 2021-05-04T00:55:35.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8eb8234-ea3a-4123-b437-cfe1787bff44 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:35.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8eb8234-ea3a-4123-b437-cfe1787bff44 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:38.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=504bdd1a-30ae-420f-97de-ed866d6dc0f3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:38.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=504bdd1a-30ae-420f-97de-ed866d6dc0f3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54c2eba8 2021-05-04T00:55:38.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=504bdd1a-30ae-420f-97de-ed866d6dc0f3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:38.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=504bdd1a-30ae-420f-97de-ed866d6dc0f3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:41.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49e6d2f4-3073-458b-8e06-50f15fa5633c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:41.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49e6d2f4-3073-458b-8e06-50f15fa5633c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e3efc08 2021-05-04T00:55:41.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49e6d2f4-3073-458b-8e06-50f15fa5633c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:41.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49e6d2f4-3073-458b-8e06-50f15fa5633c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:44.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f13bd995-2c05-402b-9e16-70973ab12da9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:44.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f13bd995-2c05-402b-9e16-70973ab12da9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53e7e213 2021-05-04T00:55:44.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f13bd995-2c05-402b-9e16-70973ab12da9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:44.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f13bd995-2c05-402b-9e16-70973ab12da9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:47.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=628a8972-940d-4045-a9d3-f3ab1968e295 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:47.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=628a8972-940d-4045-a9d3-f3ab1968e295 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d724619 2021-05-04T00:55:47.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=628a8972-940d-4045-a9d3-f3ab1968e295 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:47.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=628a8972-940d-4045-a9d3-f3ab1968e295 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:50.152Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=628a8972-940d-4045-a9d3-f3ab1968e295 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:55:50.153Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=628a8972-940d-4045-a9d3-f3ab1968e295 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:55:50.153Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=628a8972-940d-4045-a9d3-f3ab1968e295 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:55:50.153Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=628a8972-940d-4045-a9d3-f3ab1968e295 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:55:52.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4745e4d8-3529-4ce1-89b0-8ed2eb8b0160 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:52.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4745e4d8-3529-4ce1-89b0-8ed2eb8b0160 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1eae69b4 2021-05-04T00:55:52.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4745e4d8-3529-4ce1-89b0-8ed2eb8b0160 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:52.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4745e4d8-3529-4ce1-89b0-8ed2eb8b0160 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:52.656Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4745e4d8-3529-4ce1-89b0-8ed2eb8b0160 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:55:52.656Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4745e4d8-3529-4ce1-89b0-8ed2eb8b0160 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:55:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb23dbbd-a641-4d4e-8a63-b633410fa52d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb23dbbd-a641-4d4e-8a63-b633410fa52d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b429347 2021-05-04T00:55:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb23dbbd-a641-4d4e-8a63-b633410fa52d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb23dbbd-a641-4d4e-8a63-b633410fa52d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:56.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ddc0056-c0da-4fc3-b4f8-c13d1ca265f7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:56.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ddc0056-c0da-4fc3-b4f8-c13d1ca265f7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@144f54b8 2021-05-04T00:55:56.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ddc0056-c0da-4fc3-b4f8-c13d1ca265f7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:56.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ddc0056-c0da-4fc3-b4f8-c13d1ca265f7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:55:59.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef675f3a-5c56-432c-9e8f-ccbc6ee8e49b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:55:59.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef675f3a-5c56-432c-9e8f-ccbc6ee8e49b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79417c4c 2021-05-04T00:55:59.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef675f3a-5c56-432c-9e8f-ccbc6ee8e49b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:55:59.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef675f3a-5c56-432c-9e8f-ccbc6ee8e49b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:02.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f27da9df-89a1-4f0e-94fa-763d67c309db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:02.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f27da9df-89a1-4f0e-94fa-763d67c309db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a1cc01 2021-05-04T00:56:02.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f27da9df-89a1-4f0e-94fa-763d67c309db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:02.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f27da9df-89a1-4f0e-94fa-763d67c309db PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:05.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e09b795d-1087-4391-88ae-48637b9ff940 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:05.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e09b795d-1087-4391-88ae-48637b9ff940 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4936bf39 2021-05-04T00:56:05.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e09b795d-1087-4391-88ae-48637b9ff940 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:05.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e09b795d-1087-4391-88ae-48637b9ff940 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf84bfb-89ea-4fbb-908e-29de922d3e41 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf84bfb-89ea-4fbb-908e-29de922d3e41 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27b2c833 2021-05-04T00:56:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf84bfb-89ea-4fbb-908e-29de922d3e41 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf84bfb-89ea-4fbb-908e-29de922d3e41 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T00:56:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T00:56:11.213Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5cf84bfb-89ea-4fbb-908e-29de922d3e41 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:56:11.214Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5cf84bfb-89ea-4fbb-908e-29de922d3e41 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:56:11.214Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5cf84bfb-89ea-4fbb-908e-29de922d3e41 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:56:11.214Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5cf84bfb-89ea-4fbb-908e-29de922d3e41 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:56:11.857Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T00:56:12.855Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69f74086-5ba9-4587-aa3a-e8ddeddca1aa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:12.855Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69f74086-5ba9-4587-aa3a-e8ddeddca1aa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55e37395 2021-05-04T00:56:12.855Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69f74086-5ba9-4587-aa3a-e8ddeddca1aa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:12.855Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69f74086-5ba9-4587-aa3a-e8ddeddca1aa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:13.404Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=69f74086-5ba9-4587-aa3a-e8ddeddca1aa ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:56:13.404Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=69f74086-5ba9-4587-aa3a-e8ddeddca1aa ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T00:56:14.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd9876c2-9287-44f8-963d-61d6b09a86e4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:14.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd9876c2-9287-44f8-963d-61d6b09a86e4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2119cf07 2021-05-04T00:56:14.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd9876c2-9287-44f8-963d-61d6b09a86e4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:14.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd9876c2-9287-44f8-963d-61d6b09a86e4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:14.965Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cd9876c2-9287-44f8-963d-61d6b09a86e4 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:56:14.965Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cd9876c2-9287-44f8-963d-61d6b09a86e4 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:56:17.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b48b5044-09fd-46a0-b3c8-989508a3fa05 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:17.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b48b5044-09fd-46a0-b3c8-989508a3fa05 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@105b6847 2021-05-04T00:56:17.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b48b5044-09fd-46a0-b3c8-989508a3fa05 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:17.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b48b5044-09fd-46a0-b3c8-989508a3fa05 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:20.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0473c94c-edd0-4d0f-9aad-2d87bcf6124c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:20.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0473c94c-edd0-4d0f-9aad-2d87bcf6124c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5eeade02 2021-05-04T00:56:20.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0473c94c-edd0-4d0f-9aad-2d87bcf6124c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:20.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0473c94c-edd0-4d0f-9aad-2d87bcf6124c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:23.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00b92299-056e-4f78-ab56-c644063a12dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:23.607Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00b92299-056e-4f78-ab56-c644063a12dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5029943f 2021-05-04T00:56:23.607Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00b92299-056e-4f78-ab56-c644063a12dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:23.607Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00b92299-056e-4f78-ab56-c644063a12dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:26.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ed904d3-cf0e-4a66-bbbd-e5cfb6b787ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:26.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ed904d3-cf0e-4a66-bbbd-e5cfb6b787ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3fa718ca 2021-05-04T00:56:26.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ed904d3-cf0e-4a66-bbbd-e5cfb6b787ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:26.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ed904d3-cf0e-4a66-bbbd-e5cfb6b787ba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:29.233Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f5f3870-4c21-4eec-96c3-f5a1b72613b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:29.233Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f5f3870-4c21-4eec-96c3-f5a1b72613b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74f55629 2021-05-04T00:56:29.233Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f5f3870-4c21-4eec-96c3-f5a1b72613b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:29.233Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f5f3870-4c21-4eec-96c3-f5a1b72613b8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:32.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6f75753-0d0e-4af7-8edb-59609c07287b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:32.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6f75753-0d0e-4af7-8edb-59609c07287b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ce78c7b 2021-05-04T00:56:32.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6f75753-0d0e-4af7-8edb-59609c07287b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:32.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6f75753-0d0e-4af7-8edb-59609c07287b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc9a27f3-9f09-4cc5-b2f0-95ac260b5908 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc9a27f3-9f09-4cc5-b2f0-95ac260b5908 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60f367b4 2021-05-04T00:56:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc9a27f3-9f09-4cc5-b2f0-95ac260b5908 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc9a27f3-9f09-4cc5-b2f0-95ac260b5908 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:38.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8236dee6-134e-4ac1-86b0-15e761e9ab5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:38.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8236dee6-134e-4ac1-86b0-15e761e9ab5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67955228 2021-05-04T00:56:38.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8236dee6-134e-4ac1-86b0-15e761e9ab5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:38.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8236dee6-134e-4ac1-86b0-15e761e9ab5f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc0e0e7-5bbe-400c-a4b2-ffc3540d26c5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc0e0e7-5bbe-400c-a4b2-ffc3540d26c5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@390f14ed 2021-05-04T00:56:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc0e0e7-5bbe-400c-a4b2-ffc3540d26c5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc0e0e7-5bbe-400c-a4b2-ffc3540d26c5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:44.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4699d454-ab94-4770-b970-7e4b4d47397e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:44.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4699d454-ab94-4770-b970-7e4b4d47397e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79cbc3e3 2021-05-04T00:56:44.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4699d454-ab94-4770-b970-7e4b4d47397e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:44.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4699d454-ab94-4770-b970-7e4b4d47397e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:47.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afb1895a-808a-4986-98f7-b86bb7dcdb25 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:47.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afb1895a-808a-4986-98f7-b86bb7dcdb25 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27ec5a9 2021-05-04T00:56:47.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afb1895a-808a-4986-98f7-b86bb7dcdb25 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:47.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afb1895a-808a-4986-98f7-b86bb7dcdb25 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:51.685Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1ad713c-4dcd-48a1-a791-ec09a2525395 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:51.685Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1ad713c-4dcd-48a1-a791-ec09a2525395 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f37d3dc 2021-05-04T00:56:51.685Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1ad713c-4dcd-48a1-a791-ec09a2525395 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:51.685Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1ad713c-4dcd-48a1-a791-ec09a2525395 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af2b1ac5-7b4e-4899-a072-a8726450eaee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af2b1ac5-7b4e-4899-a072-a8726450eaee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ede0eab 2021-05-04T00:56:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af2b1ac5-7b4e-4899-a072-a8726450eaee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af2b1ac5-7b4e-4899-a072-a8726450eaee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:56.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44b5b893-8cbf-42be-9130-28f5d3797c0b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:56.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44b5b893-8cbf-42be-9130-28f5d3797c0b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@292d0048 2021-05-04T00:56:56.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44b5b893-8cbf-42be-9130-28f5d3797c0b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:56.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44b5b893-8cbf-42be-9130-28f5d3797c0b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:56:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1171d558-368c-4302-acca-ae23c301c064 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:56:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1171d558-368c-4302-acca-ae23c301c064 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d8adb49 2021-05-04T00:56:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1171d558-368c-4302-acca-ae23c301c064 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:56:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1171d558-368c-4302-acca-ae23c301c064 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:02.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aabe622b-6ce7-44aa-aad5-ad94fb19d522 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:02.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aabe622b-6ce7-44aa-aad5-ad94fb19d522 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cd25a17 2021-05-04T00:57:02.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aabe622b-6ce7-44aa-aad5-ad94fb19d522 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:02.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aabe622b-6ce7-44aa-aad5-ad94fb19d522 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:05.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c09c62f1-f9c2-43e2-89be-a0bfb29559a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:05.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c09c62f1-f9c2-43e2-89be-a0bfb29559a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5287a270 2021-05-04T00:57:05.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c09c62f1-f9c2-43e2-89be-a0bfb29559a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:05.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c09c62f1-f9c2-43e2-89be-a0bfb29559a2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:08.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=518e4c59-7bb0-4050-a52e-6c78b8b9325b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:08.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=518e4c59-7bb0-4050-a52e-6c78b8b9325b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13e9d728 2021-05-04T00:57:08.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=518e4c59-7bb0-4050-a52e-6c78b8b9325b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:08.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=518e4c59-7bb0-4050-a52e-6c78b8b9325b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:11.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4794965b-d266-4054-a148-1a120d3a4703 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:11.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4794965b-d266-4054-a148-1a120d3a4703 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1832068f 2021-05-04T00:57:11.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4794965b-d266-4054-a148-1a120d3a4703 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:11.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4794965b-d266-4054-a148-1a120d3a4703 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd058d83-3b7a-43af-8dc0-33240a11ffdc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd058d83-3b7a-43af-8dc0-33240a11ffdc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@268ee57c 2021-05-04T00:57:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd058d83-3b7a-43af-8dc0-33240a11ffdc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd058d83-3b7a-43af-8dc0-33240a11ffdc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:17.793Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f4bcc0d-437f-4ddf-927e-ed09f587e5b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:17.793Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f4bcc0d-437f-4ddf-927e-ed09f587e5b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49f547ff 2021-05-04T00:57:17.793Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f4bcc0d-437f-4ddf-927e-ed09f587e5b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:17.793Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f4bcc0d-437f-4ddf-927e-ed09f587e5b8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=869663f5-5f4d-4b5d-b83b-2f7172d287ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=869663f5-5f4d-4b5d-b83b-2f7172d287ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1478897e 2021-05-04T00:57:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=869663f5-5f4d-4b5d-b83b-2f7172d287ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=869663f5-5f4d-4b5d-b83b-2f7172d287ba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:23.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d44850da-e064-4072-ab21-86944d6967a8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:23.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d44850da-e064-4072-ab21-86944d6967a8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1cf6e300 2021-05-04T00:57:23.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d44850da-e064-4072-ab21-86944d6967a8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:23.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d44850da-e064-4072-ab21-86944d6967a8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:29.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=402345f0-9f91-428a-8d27-3d1e51fcdbcb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:29.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=402345f0-9f91-428a-8d27-3d1e51fcdbcb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d8240e7 2021-05-04T00:57:29.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=402345f0-9f91-428a-8d27-3d1e51fcdbcb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:29.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=402345f0-9f91-428a-8d27-3d1e51fcdbcb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:30.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ace531d-4ec2-400e-bbce-4cdecefc8f83 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:30.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ace531d-4ec2-400e-bbce-4cdecefc8f83 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2eab2c17 2021-05-04T00:57:30.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ace531d-4ec2-400e-bbce-4cdecefc8f83 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:30.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ace531d-4ec2-400e-bbce-4cdecefc8f83 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:32.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6880fa31-2319-4a47-a582-b7d49335dd01 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:32.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6880fa31-2319-4a47-a582-b7d49335dd01 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ec863c5 2021-05-04T00:57:32.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6880fa31-2319-4a47-a582-b7d49335dd01 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:32.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6880fa31-2319-4a47-a582-b7d49335dd01 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:35.683Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4534c7e1-11e2-41df-9320-e14783c4ada2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:35.683Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4534c7e1-11e2-41df-9320-e14783c4ada2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@558d61ff 2021-05-04T00:57:35.683Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4534c7e1-11e2-41df-9320-e14783c4ada2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:35.683Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4534c7e1-11e2-41df-9320-e14783c4ada2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:38.448Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0566f110-caaa-4882-b5d7-63c1ed885589 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:38.448Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0566f110-caaa-4882-b5d7-63c1ed885589 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ad786d1 2021-05-04T00:57:38.448Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0566f110-caaa-4882-b5d7-63c1ed885589 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:38.448Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0566f110-caaa-4882-b5d7-63c1ed885589 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:41.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaa30298-9b9d-460b-8f65-cf5c74cf834b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:41.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaa30298-9b9d-460b-8f65-cf5c74cf834b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7d6bd808 2021-05-04T00:57:41.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaa30298-9b9d-460b-8f65-cf5c74cf834b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:41.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaa30298-9b9d-460b-8f65-cf5c74cf834b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9021905-811a-4651-a382-a827abd03ee6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9021905-811a-4651-a382-a827abd03ee6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40804a2b 2021-05-04T00:57:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9021905-811a-4651-a382-a827abd03ee6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9021905-811a-4651-a382-a827abd03ee6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:46.952Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0412f5a2-007c-4f41-a44a-7af0d2f4f3d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:46.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0412f5a2-007c-4f41-a44a-7af0d2f4f3d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ebc595d 2021-05-04T00:57:46.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0412f5a2-007c-4f41-a44a-7af0d2f4f3d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:46.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0412f5a2-007c-4f41-a44a-7af0d2f4f3d7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:50.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4b44929-66e1-4775-81a1-2dace68cada4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:50.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4b44929-66e1-4775-81a1-2dace68cada4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48eba955 2021-05-04T00:57:50.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4b44929-66e1-4775-81a1-2dace68cada4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:50.120Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4b44929-66e1-4775-81a1-2dace68cada4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:50.267Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Read timed out; blacklisting for 2 minutes 2021-05-04T00:57:50.272Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.276Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.276Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.278Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.278Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.279Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.279Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.280Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.281Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.282Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.282Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.283Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.283Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.284Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.284Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.285Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.286Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.289Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.289Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.290Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.290Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.292Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.292Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.293Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.293Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.295Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.295Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.296Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.296Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.297Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:57:50.297Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.298Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.299Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.300Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.300Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.301Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.301Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.306Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.306Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.307Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.307Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.309Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.309Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.310Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.310Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.311Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.311Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.312Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.313Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.314Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.314Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.315Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.315Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.316Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.316Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.317Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.318Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.319Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.319Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.320Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.320Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.321Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.321Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.322Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.323Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.324Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.324Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.325Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.325Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.326Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.326Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.327Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.328Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.329Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.329Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.331Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.331Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.332Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.332Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.333Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.334Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.335Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.335Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.336Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.336Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.337Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.337Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.338Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.339Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.340Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.340Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.341Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.341Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.342Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.342Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.343Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.344Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.345Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.345Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.346Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.346Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.348Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:57:50.348Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.349Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.351Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.352Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.352Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.354Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.354Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.356Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.356Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.357Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.357Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.359Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.359Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.361Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.361Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.363Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.363Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.364Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.365Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.367Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.367Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.368Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.368Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.370Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:57:50.370Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.371Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.371Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.373Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.373Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.374Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.374Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.375Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.375Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.376Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.377Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.379Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.379Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.380Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.380Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.381Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.381Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.382Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.383Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.384Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.384Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.385Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.385Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.386Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.386Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.387Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.388Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.389Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.390Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.391Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.391Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.392Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.392Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.393Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.394Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.395Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.395Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.396Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.396Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.397Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.397Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.398Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.398Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.400Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.400Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.401Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:57:50.401Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.403Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.403Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.404Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.405Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.406Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.406Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.408Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.408Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.409Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.409Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.410Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.411Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.412Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.412Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.414Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.414Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.416Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.416Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.417Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.418Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.420Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.420Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.421Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.421Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.423Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.423Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.424Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.425Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.426Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:57:50.426Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.428Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.428Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.429Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.429Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.430Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.431Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.440Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.440Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.442Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.442Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.443Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.443Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.445Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.446Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.447Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.447Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.448Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.448Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.449Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.449Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.451Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.452Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.453Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.453Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.454Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.454Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.456Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.456Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.457Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.458Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.459Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.459Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.461Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.461Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.462Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.462Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.463Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.464Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.465Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.466Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.467Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.467Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.468Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.468Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.469Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.470Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.472Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.472Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.473Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.473Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.474Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.474Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.475Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.476Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.478Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.478Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.479Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.479Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.481Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.481Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.482Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.483Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.485Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.485Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.486Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.486Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.487Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.487Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.489Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.489Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.491Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.491Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.492Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.492Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.494Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.494Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.495Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.496Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.498Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.498Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.499Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.499Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.500Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.500Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.502Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.502Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.504Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.504Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.505Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.505Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.506Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.506Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.507Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.508Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.510Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.510Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.511Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.511Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.512Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.512Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.514Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.515Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.516Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.516Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.517Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.517Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.519Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.519Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.520Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.521Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.522Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.523Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.524Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.524Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.525Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.525Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.527Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T00:57:50.528Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.529Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.529Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.531Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.531Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.532Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.532Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.533Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.534Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.535Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.535Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.536Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.537Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.538Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.538Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.539Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.540Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.541Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.541Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.542Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.542Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.544Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.544Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.545Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.546Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.547Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.547Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.548Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.548Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.549Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.549Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.551Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.551Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.553Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.553Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.554Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.554Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.555Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.555Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.556Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.557Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.558Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.558Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.560Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.560Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.561Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.561Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.562Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.563Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.564Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.564Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.565Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.565Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.566Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.566Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.567Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.568Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.569Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.569Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.570Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:57:50.570Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.571Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.572Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.573Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.573Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.575Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.575Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.576Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.576Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.577Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.577Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.578Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.579Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.581Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.581Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.582Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.582Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.583Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.583Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.584Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.585Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.586Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.586Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.587Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.587Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.588Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.588Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.590Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.590Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.592Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.592Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.593Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.593Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.594Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.594Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.595Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.596Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.597Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.598Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.599Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.599Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.600Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.600Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.601Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.602Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.603Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.603Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.604Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.604Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.606Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.606Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.607Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.608Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.609Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.609Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.610Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.610Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.611Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.611Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.612Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.613Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.614Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.614Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.615Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.615Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.616Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.616Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.617Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.618Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.619Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.619Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.620Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.620Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.621Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.621Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.622Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.623Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.624Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.624Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.625Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.625Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.626Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:57:50.626Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.627Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.628Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.629Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.630Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.631Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.631Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.632Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:57:50.632Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.633Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.634Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.635Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.635Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.636Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.636Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.638Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.638Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.642Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T00:57:50.643Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.644Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.644Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.646Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.646Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.647Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.647Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.648Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.649Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.650Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.650Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.651Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.651Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.652Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.653Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.654Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.654Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.656Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.656Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.657Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.657Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.658Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.658Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.659Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.660Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.661Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.661Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.663Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.663Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.664Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.664Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.665Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.666Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.667Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.667Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.668Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.668Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.669Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.669Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.671Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.671Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.673Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.673Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.674Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.674Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.675Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.675Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.677Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.677Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.679Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.679Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.680Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.680Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.681Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.681Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.683Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.683Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.685Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.685Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.686Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.686Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.688Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.688Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.689Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.690Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.691Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.691Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.692Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.692Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.695Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.695Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.696Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T00:57:50.697Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.698Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.698Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.699Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.699Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.700Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.700Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.701Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.702Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.703Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.703Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.705Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.705Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.706Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.706Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.707Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.707Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.709Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.709Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.710Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.710Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.711Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.711Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.712Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.713Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.714Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.714Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.715Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.715Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.716Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.716Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.717Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.718Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.719Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.719Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.720Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.720Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.722Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.722Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.723Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.723Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.725Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.725Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.726Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.726Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.727Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.727Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.728Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.728Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.729Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.729Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.731Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.731Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.732Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.732Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.733Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.734Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.735Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.735Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.736Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.736Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.738Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.738Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.739Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.739Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.741Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.741Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.742Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.742Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.743Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.743Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.745Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.745Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.747Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.747Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.748Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.748Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.749Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.749Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.750Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.751Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.752Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.752Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.753Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.753Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.754Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.754Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.755Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:50.757Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:57:50.759Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.760Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.761Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.761Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.762Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:57:50.762Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:57:50.763Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:57:53.527Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0863a4c1-98bf-4c53-b53b-b870b95d041b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:53.527Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0863a4c1-98bf-4c53-b53b-b870b95d041b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b6393d8 2021-05-04T00:57:53.527Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0863a4c1-98bf-4c53-b53b-b870b95d041b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:53.527Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0863a4c1-98bf-4c53-b53b-b870b95d041b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6d87aba-cb0c-4938-ad99-9e136ded1fcc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6d87aba-cb0c-4938-ad99-9e136ded1fcc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29f05d8d 2021-05-04T00:57:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6d87aba-cb0c-4938-ad99-9e136ded1fcc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6d87aba-cb0c-4938-ad99-9e136ded1fcc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:57:59.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2f092d-5609-40f2-8f32-c3e1d7b46a4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:57:59.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2f092d-5609-40f2-8f32-c3e1d7b46a4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@359a9f5f 2021-05-04T00:57:59.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2f092d-5609-40f2-8f32-c3e1d7b46a4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:57:59.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2f092d-5609-40f2-8f32-c3e1d7b46a4a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:02.955Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=9c2f092d-5609-40f2-8f32-c3e1d7b46a4a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:58:02.955Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=9c2f092d-5609-40f2-8f32-c3e1d7b46a4a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:58:02.956Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9c2f092d-5609-40f2-8f32-c3e1d7b46a4a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:58:02.956Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9c2f092d-5609-40f2-8f32-c3e1d7b46a4a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:58:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:58:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:05.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:05.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:58:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:05.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:08.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93f732dd-6fba-4b97-8b19-edd494913113 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:08.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93f732dd-6fba-4b97-8b19-edd494913113 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@202e39b8 2021-05-04T00:58:08.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93f732dd-6fba-4b97-8b19-edd494913113 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:08.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93f732dd-6fba-4b97-8b19-edd494913113 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:11.372Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=93f732dd-6fba-4b97-8b19-edd494913113 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:58:11.372Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=93f732dd-6fba-4b97-8b19-edd494913113 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:58:12.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4ce812a-001a-46bd-8f19-7ee0716b2002 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:12.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4ce812a-001a-46bd-8f19-7ee0716b2002 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c13b07a 2021-05-04T00:58:12.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4ce812a-001a-46bd-8f19-7ee0716b2002 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:12.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4ce812a-001a-46bd-8f19-7ee0716b2002 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:14.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49e8c4e-81ca-4cac-8ab8-d006534c42ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:14.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49e8c4e-81ca-4cac-8ab8-d006534c42ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d92a8c0 2021-05-04T00:58:14.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49e8c4e-81ca-4cac-8ab8-d006534c42ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:14.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49e8c4e-81ca-4cac-8ab8-d006534c42ba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:15.719Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce28fc32-9aec-4d41-b3c5-16aba1f4e142 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:15.719Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce28fc32-9aec-4d41-b3c5-16aba1f4e142 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c20b133 2021-05-04T00:58:15.719Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce28fc32-9aec-4d41-b3c5-16aba1f4e142 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:15.719Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce28fc32-9aec-4d41-b3c5-16aba1f4e142 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:16.615Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15af7c7a-cdab-416c-ba21-5ddcf80d62ce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:16.615Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15af7c7a-cdab-416c-ba21-5ddcf80d62ce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f6ea6a2 2021-05-04T00:58:16.615Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15af7c7a-cdab-416c-ba21-5ddcf80d62ce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:16.615Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15af7c7a-cdab-416c-ba21-5ddcf80d62ce PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:17.831Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cfcdf9b4-56de-43ae-9ede-5c5f60f33ab8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:17.831Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cfcdf9b4-56de-43ae-9ede-5c5f60f33ab8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e9f5155 2021-05-04T00:58:17.832Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cfcdf9b4-56de-43ae-9ede-5c5f60f33ab8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:17.832Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cfcdf9b4-56de-43ae-9ede-5c5f60f33ab8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:58:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:58:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:20.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:20.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:58:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:58:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:20.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:20.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0258e758-23d1-42b9-9ef0-7fec7f30e0eb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:20.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0258e758-23d1-42b9-9ef0-7fec7f30e0eb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56c326ee 2021-05-04T00:58:20.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0258e758-23d1-42b9-9ef0-7fec7f30e0eb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:20.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0258e758-23d1-42b9-9ef0-7fec7f30e0eb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecc58f63-67db-44fc-aa13-d50d5b5601fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecc58f63-67db-44fc-aa13-d50d5b5601fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f23bc2c 2021-05-04T00:58:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecc58f63-67db-44fc-aa13-d50d5b5601fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecc58f63-67db-44fc-aa13-d50d5b5601fe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:26.509Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=ecc58f63-67db-44fc-aa13-d50d5b5601fe ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:58:26.511Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=ecc58f63-67db-44fc-aa13-d50d5b5601fe ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:58:26.511Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ecc58f63-67db-44fc-aa13-d50d5b5601fe ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:58:26.511Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ecc58f63-67db-44fc-aa13-d50d5b5601fe ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:58:26.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec6d0490-bc48-4543-bd6b-fcc8e6fb3d36 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:26.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec6d0490-bc48-4543-bd6b-fcc8e6fb3d36 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@553242dc 2021-05-04T00:58:26.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec6d0490-bc48-4543-bd6b-fcc8e6fb3d36 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:26.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec6d0490-bc48-4543-bd6b-fcc8e6fb3d36 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:29.918Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=ec6d0490-bc48-4543-bd6b-fcc8e6fb3d36 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T00:58:29.918Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=ec6d0490-bc48-4543-bd6b-fcc8e6fb3d36 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T00:58:30.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3918a89-97fd-4a5b-ac0f-f22a9acb07f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:30.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3918a89-97fd-4a5b-ac0f-f22a9acb07f8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3eb8185e 2021-05-04T00:58:30.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3918a89-97fd-4a5b-ac0f-f22a9acb07f8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:30.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3918a89-97fd-4a5b-ac0f-f22a9acb07f8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:31.860Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c3918a89-97fd-4a5b-ac0f-f22a9acb07f8 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:58:31.860Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c3918a89-97fd-4a5b-ac0f-f22a9acb07f8 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:58:32.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=852605ac-3f64-48bc-b3ae-82fa166891d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:32.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=852605ac-3f64-48bc-b3ae-82fa166891d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b9b9098 2021-05-04T00:58:32.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=852605ac-3f64-48bc-b3ae-82fa166891d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:32.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=852605ac-3f64-48bc-b3ae-82fa166891d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:58:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:35.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:35.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:35.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:35.182Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:58:35.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:35.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:35.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:35.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:35.188Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:35.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e8a32d0-138f-4bf7-b26d-c37e0fe96c3a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:35.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e8a32d0-138f-4bf7-b26d-c37e0fe96c3a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17d2be39 2021-05-04T00:58:35.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e8a32d0-138f-4bf7-b26d-c37e0fe96c3a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:35.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e8a32d0-138f-4bf7-b26d-c37e0fe96c3a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:37.211Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Read timed out; blacklisting for 2 minutes 2021-05-04T00:58:37.212Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.212Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.212Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.212Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.213Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.213Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.216Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.216Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.224Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.225Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.230Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:58:37.230Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.232Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.232Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.232Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.232Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.232Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.233Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.233Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.234Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:58:37.234Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.236Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.236Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.237Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.237Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.238Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.239Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.239Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.239Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.239Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.239Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.239Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.240Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.240Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.242Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.242Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.243Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.243Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.244Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.245Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.245Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.245Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.245Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.245Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.245Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.246Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:58:37.247Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.248Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.248Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.249Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:58:37.249Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.250Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.251Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.251Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.251Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.251Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.251Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.251Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.252Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.252Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.254Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.254Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.255Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.255Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.256Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.256Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.256Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.256Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.256Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.257Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.257Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.258Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.258Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.259Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.259Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.261Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.261Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.262Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.262Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.262Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.262Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.262Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.263Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.263Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.264Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.264Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.265Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.265Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.266Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.266Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.267Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.268Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.268Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.268Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.268Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.268Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.268Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.269Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.269Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.271Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.271Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.272Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.272Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.273Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.273Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.274Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.274Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.274Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.274Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.274Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.275Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.275Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.276Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.277Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.278Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.278Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.279Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.279Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.279Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:37.280Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:58:37.280Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:58:37.280Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:58:37.280Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:58:37.281Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.281Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.282Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:37.282Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.285Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T00:58:37.285Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:37.286Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:37.286Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T00:58:37.286Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T00:58:38.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e34e0f0-1245-4a47-84f9-7a7a6414ee5b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:38.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e34e0f0-1245-4a47-84f9-7a7a6414ee5b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69d62e75 2021-05-04T00:58:38.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e34e0f0-1245-4a47-84f9-7a7a6414ee5b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:38.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e34e0f0-1245-4a47-84f9-7a7a6414ee5b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:41.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ddcd90fa-1ae7-42f3-a78a-a74e63914c98 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:41.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ddcd90fa-1ae7-42f3-a78a-a74e63914c98 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@155989ef 2021-05-04T00:58:41.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ddcd90fa-1ae7-42f3-a78a-a74e63914c98 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:41.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ddcd90fa-1ae7-42f3-a78a-a74e63914c98 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:44.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20352e3b-21de-4565-9fb4-c20b3af755dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:44.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20352e3b-21de-4565-9fb4-c20b3af755dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32e1d48c 2021-05-04T00:58:44.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20352e3b-21de-4565-9fb4-c20b3af755dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:44.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20352e3b-21de-4565-9fb4-c20b3af755dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:47.249Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91e8202e-9421-480d-94b9-2429376b867d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:47.249Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91e8202e-9421-480d-94b9-2429376b867d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3040879f 2021-05-04T00:58:47.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91e8202e-9421-480d-94b9-2429376b867d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:47.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91e8202e-9421-480d-94b9-2429376b867d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:58:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:50.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:50.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:58:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:58:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:58:50.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:58:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03824cb-bd9c-4d85-8e5d-7e080db77950 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03824cb-bd9c-4d85-8e5d-7e080db77950 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@683db264 2021-05-04T00:58:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03824cb-bd9c-4d85-8e5d-7e080db77950 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f03824cb-bd9c-4d85-8e5d-7e080db77950 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:51.257Z [qtp580718781-17] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=34fa1726-cba5-4812-9488-b930c79a5fbc InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:58:51.256878Z InvocationID=a92d87bd-8d77-4af6-8ea1-4fc8ea46d460 LogTimestamp=2021-05-04T00:58:51.257058Z Entering 2021-05-04T00:58:51.258Z [qtp580718781-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=34fa1726-cba5-4812-9488-b930c79a5fbc InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:58:51.256878Z InvocationID=a92d87bd-8d77-4af6-8ea1-4fc8ea46d460 LogTimestamp=2021-05-04T00:58:51.257058Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:58:51.262Z [qtp580718781-17] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=3 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:58:51.258Z InvocationID=b85eef47-7b27-45d5-926b-1c16703fb716 LogTimestamp=2021-05-04T00:58:51.261956Z ResponseDesc=OK Exiting. 2021-05-04T00:58:51.262Z [qtp580718781-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:58:53.550Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f55ef5b3-9752-4d72-95ca-b6f2ac5e1723 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:53.550Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f55ef5b3-9752-4d72-95ca-b6f2ac5e1723 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3172587d 2021-05-04T00:58:53.550Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f55ef5b3-9752-4d72-95ca-b6f2ac5e1723 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:53.550Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f55ef5b3-9752-4d72-95ca-b6f2ac5e1723 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d73a3bec-797a-4ed4-953f-8a4206eeab1b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d73a3bec-797a-4ed4-953f-8a4206eeab1b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d0baf6b 2021-05-04T00:58:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d73a3bec-797a-4ed4-953f-8a4206eeab1b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d73a3bec-797a-4ed4-953f-8a4206eeab1b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:58:59.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d360b09-f82f-487c-afe5-6f092f2c3905 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:58:59.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d360b09-f82f-487c-afe5-6f092f2c3905 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39c3ad2a 2021-05-04T00:58:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d360b09-f82f-487c-afe5-6f092f2c3905 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:58:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d360b09-f82f-487c-afe5-6f092f2c3905 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:02.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0485c09b-a882-43a3-9b35-8bce799ed002 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:02.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0485c09b-a882-43a3-9b35-8bce799ed002 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ce886e4 2021-05-04T00:59:02.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0485c09b-a882-43a3-9b35-8bce799ed002 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:02.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0485c09b-a882-43a3-9b35-8bce799ed002 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:59:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:59:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:59:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:59:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:59:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:59:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:59:05.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:59:05.182Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:59:05.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:59:05.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:59:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:59:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:59:05.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T00:59:05.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T00:59:05.190Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T00:59:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcd7622e-16d5-413c-8fc1-bbcea42aba5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcd7622e-16d5-413c-8fc1-bbcea42aba5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ffa80b4 2021-05-04T00:59:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcd7622e-16d5-413c-8fc1-bbcea42aba5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcd7622e-16d5-413c-8fc1-bbcea42aba5f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96076c6-c765-4679-9bc0-c864fa725f4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96076c6-c765-4679-9bc0-c864fa725f4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4fda14d7 2021-05-04T00:59:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96076c6-c765-4679-9bc0-c864fa725f4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96076c6-c765-4679-9bc0-c864fa725f4a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:11.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b146446d-733d-4e9b-b325-e191eda9c445 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:11.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b146446d-733d-4e9b-b325-e191eda9c445 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6bd45afb 2021-05-04T00:59:11.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b146446d-733d-4e9b-b325-e191eda9c445 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:11.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b146446d-733d-4e9b-b325-e191eda9c445 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:14.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d72b82-48d3-450f-ba03-1202b379022d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:14.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d72b82-48d3-450f-ba03-1202b379022d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13732ec3 2021-05-04T00:59:14.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d72b82-48d3-450f-ba03-1202b379022d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:14.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d72b82-48d3-450f-ba03-1202b379022d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:17.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a271e14e-0782-422d-ac82-939c2c8e1100 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:17.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a271e14e-0782-422d-ac82-939c2c8e1100 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58c7f339 2021-05-04T00:59:17.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a271e14e-0782-422d-ac82-939c2c8e1100 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:17.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a271e14e-0782-422d-ac82-939c2c8e1100 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:20.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=285724cb-526b-489b-b3c3-e57e2908a28b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:20.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=285724cb-526b-489b-b3c3-e57e2908a28b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47a4285a 2021-05-04T00:59:20.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=285724cb-526b-489b-b3c3-e57e2908a28b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:20.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=285724cb-526b-489b-b3c3-e57e2908a28b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:59:20.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:59:20.175Z [UEB-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend connection recovery to UEB Cluster. 2021-05-04T00:59:23.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93bc867f-d6f7-4530-969b-1f459644f6f3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:23.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93bc867f-d6f7-4530-969b-1f459644f6f3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e6ea1d7 2021-05-04T00:59:23.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93bc867f-d6f7-4530-969b-1f459644f6f3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:23.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93bc867f-d6f7-4530-969b-1f459644f6f3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:23.759Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=93bc867f-d6f7-4530-969b-1f459644f6f3 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:59:23.759Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=93bc867f-d6f7-4530-969b-1f459644f6f3 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T00:59:25.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15bbca28-a675-43bf-8fec-ecdf3bcd9ab7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:25.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15bbca28-a675-43bf-8fec-ecdf3bcd9ab7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e183ef 2021-05-04T00:59:25.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15bbca28-a675-43bf-8fec-ecdf3bcd9ab7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:25.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15bbca28-a675-43bf-8fec-ecdf3bcd9ab7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:29.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce4aabf9-e21f-48c5-856d-741678b11423 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:29.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce4aabf9-e21f-48c5-856d-741678b11423 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63d8e671 2021-05-04T00:59:29.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce4aabf9-e21f-48c5-856d-741678b11423 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:29.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce4aabf9-e21f-48c5-856d-741678b11423 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:30.909Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=57f2f1ac-8cc4-4e26-b9ed-223f4bb51354 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:59:30.908328Z InvocationID=e58b4b43-5d7e-4019-8060-a47c70f30cf4 LogTimestamp=2021-05-04T00:59:30.908801Z Entering 2021-05-04T00:59:30.909Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=57f2f1ac-8cc4-4e26-b9ed-223f4bb51354 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T00:59:30.908328Z InvocationID=e58b4b43-5d7e-4019-8060-a47c70f30cf4 LogTimestamp=2021-05-04T00:59:30.908801Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T00:59:30.913Z [qtp580718781-12] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=3 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T00:59:30.910Z InvocationID=1da48146-5f76-4a0e-bac9-af6a458859dc LogTimestamp=2021-05-04T00:59:30.913251Z ResponseDesc=OK Exiting. 2021-05-04T00:59:30.914Z [qtp580718781-12] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T00:59:31.022Z [qtp580718781-16] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=4d8a7c17-a3e2-4d92-b26a-242ceef5fe37 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:59:31.022553Z InvocationID=46cca7a2-bd3c-4f5d-be96-f40b301d12c5 LogTimestamp=2021-05-04T00:59:31.022652Z Entering 2021-05-04T00:59:31.022Z [qtp580718781-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=4d8a7c17-a3e2-4d92-b26a-242ceef5fe37 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=cds auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.135 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.135 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T00:59:31.022553Z InvocationID=46cca7a2-bd3c-4f5d-be96-f40b301d12c5 LogTimestamp=2021-05-04T00:59:31.022652Z localAddr=10.242.235.154 remoteAddr=10.242.8.135 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T00:59:31.028Z [qtp580718781-16] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:59:31.023Z InvocationID=c7cbdaed-df52-4d70-b91b-f40c81ecf584 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/ajgAfPSTfRXvpiZo will send credentials over a clear channel. 2021-05-04T00:59:31.028Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:59:31.023Z InvocationID=c7cbdaed-df52-4d70-b91b-f40c81ecf584 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/ajgAfPSTfRXvpiZo (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:59:31.040Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:59:31.023Z InvocationID=c7cbdaed-df52-4d70-b91b-f40c81ecf584 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:59:31.049Z [qtp580718781-16] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "ajgAfPSTfRXvpiZo" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:59:31.023Z InvocationID=c7cbdaed-df52-4d70-b91b-f40c81ecf584 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/ajgAfPSTfRXvpiZo will send credentials over a clear channel. 2021-05-04T00:59:31.049Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "ajgAfPSTfRXvpiZo" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:59:31.023Z InvocationID=c7cbdaed-df52-4d70-b91b-f40c81ecf584 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/ajgAfPSTfRXvpiZo (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:59:31.084Z [qtp580718781-16] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "ajgAfPSTfRXvpiZo" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:59:31.023Z InvocationID=c7cbdaed-df52-4d70-b91b-f40c81ecf584 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T00:59:31.101Z [qtp580718781-16] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "cds" API_KEY = "ajgAfPSTfRXvpiZo" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=78 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T00:59:31.023Z InvocationID=c7cbdaed-df52-4d70-b91b-f40c81ecf584 LogTimestamp=2021-05-04T00:59:31.101423Z ResponseDesc=OK Exiting. 2021-05-04T00:59:31.102Z [qtp580718781-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T00:59:32.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcb6a9d7-80cd-4f8b-a262-67a768d79fda PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:32.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcb6a9d7-80cd-4f8b-a262-67a768d79fda PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@558f4a8 2021-05-04T00:59:32.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcb6a9d7-80cd-4f8b-a262-67a768d79fda PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:32.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcb6a9d7-80cd-4f8b-a262-67a768d79fda PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:59:35.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:59:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7d5857c-95de-47d8-8d40-3f24466d1ba1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:35.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7d5857c-95de-47d8-8d40-3f24466d1ba1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ec38dcd 2021-05-04T00:59:35.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7d5857c-95de-47d8-8d40-3f24466d1ba1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:35.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7d5857c-95de-47d8-8d40-3f24466d1ba1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T00:59:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T00:59:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T00:59:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T00:59:38.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51f00c96-d98a-4994-8fe4-556616664ed7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:38.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51f00c96-d98a-4994-8fe4-556616664ed7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76359701 2021-05-04T00:59:38.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51f00c96-d98a-4994-8fe4-556616664ed7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:38.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51f00c96-d98a-4994-8fe4-556616664ed7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a67704b-631d-486f-b777-f31cfd203ee9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a67704b-631d-486f-b777-f31cfd203ee9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@527a0cd8 2021-05-04T00:59:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a67704b-631d-486f-b777-f31cfd203ee9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a67704b-631d-486f-b777-f31cfd203ee9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a71a3532-0bf6-402a-964a-62c683780a6c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a71a3532-0bf6-402a-964a-62c683780a6c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10454f8b 2021-05-04T00:59:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a71a3532-0bf6-402a-964a-62c683780a6c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:44.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a71a3532-0bf6-402a-964a-62c683780a6c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:47.888Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53d99899-64c5-4c49-862f-56677e963806 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:47.889Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53d99899-64c5-4c49-862f-56677e963806 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@720a15ab 2021-05-04T00:59:47.889Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53d99899-64c5-4c49-862f-56677e963806 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:47.889Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53d99899-64c5-4c49-862f-56677e963806 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T00:59:50.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T00:59:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f2f546e-df80-4098-af88-06d2c2d01901 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f2f546e-df80-4098-af88-06d2c2d01901 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28dd0974 2021-05-04T00:59:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f2f546e-df80-4098-af88-06d2c2d01901 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:50.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f2f546e-df80-4098-af88-06d2c2d01901 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:52.051Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T00:59:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eeeb18e2-09a9-4d8c-903e-8ff35ca503c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eeeb18e2-09a9-4d8c-903e-8ff35ca503c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a43dc7b 2021-05-04T00:59:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eeeb18e2-09a9-4d8c-903e-8ff35ca503c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:53.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eeeb18e2-09a9-4d8c-903e-8ff35ca503c6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3745f4a3-fad9-470b-b39c-3dca1b13040b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3745f4a3-fad9-470b-b39c-3dca1b13040b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10c0e7af 2021-05-04T00:59:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3745f4a3-fad9-470b-b39c-3dca1b13040b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3745f4a3-fad9-470b-b39c-3dca1b13040b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T00:59:59.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10f81fe1-a20d-4aa8-820a-b7dc2317896f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T00:59:59.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10f81fe1-a20d-4aa8-820a-b7dc2317896f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32450f51 2021-05-04T00:59:59.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10f81fe1-a20d-4aa8-820a-b7dc2317896f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T00:59:59.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10f81fe1-a20d-4aa8-820a-b7dc2317896f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:03.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:00:03.203Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:00:04.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50051c4b-6e7f-45eb-aa35-6021de793e8a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:04.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50051c4b-6e7f-45eb-aa35-6021de793e8a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@721336c4 2021-05-04T01:00:04.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50051c4b-6e7f-45eb-aa35-6021de793e8a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:04.447Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50051c4b-6e7f-45eb-aa35-6021de793e8a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:00:05.189Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:00:05.603Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:00:06.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03c74c57-b65e-4ac0-98e9-08bf0991d0c4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:06.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03c74c57-b65e-4ac0-98e9-08bf0991d0c4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46078cf9 2021-05-04T01:00:06.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03c74c57-b65e-4ac0-98e9-08bf0991d0c4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:06.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03c74c57-b65e-4ac0-98e9-08bf0991d0c4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:06.545Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=03c74c57-b65e-4ac0-98e9-08bf0991d0c4 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:00:06.545Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=03c74c57-b65e-4ac0-98e9-08bf0991d0c4 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:00:08.072Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node011mf3pdntj9h8ifeukstyvq6276 2021-05-04T01:00:08.072Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0rnp9jg7kywnm178ll5vgovcon278 2021-05-04T01:00:08.072Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0l4yl33buouw519z3zrkfmalej281 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0fs7na75mgzm5ck59sgibu7zt277 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node015hntc6lc9ysn1lfb8p3mo2pn1270 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0a823vil7kjw434060ko7v0rq269 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01qf4899yzhzv7v28g1u8vyagp279 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01gaxlyyggd1m51bx0ra7pbllwc272 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01jd1bq2g6jark6wvqo5rrcsix271 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01a3jp7kelrwzstt9eeh5oyk3z274 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01epp1pjcul1w9ulfg7w20q39j273 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01k1lolwhjvigy1bfui1ug4nl7f280 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01nlxant75a5uziyzr52dt5hfb282 2021-05-04T01:00:08.073Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0g6h3ltko2rbadwmdndnjm7kv275 2021-05-04T01:00:08.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abde003a-2d07-471a-a44f-d6672dfb6960 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:08.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abde003a-2d07-471a-a44f-d6672dfb6960 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51ede609 2021-05-04T01:00:08.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abde003a-2d07-471a-a44f-d6672dfb6960 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:08.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abde003a-2d07-471a-a44f-d6672dfb6960 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:08.556Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=abde003a-2d07-471a-a44f-d6672dfb6960 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:00:08.557Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=abde003a-2d07-471a-a44f-d6672dfb6960 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:00:11.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f061796a-f67d-4702-85cd-b46057c8f7d2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:11.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f061796a-f67d-4702-85cd-b46057c8f7d2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4bc854e2 2021-05-04T01:00:11.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f061796a-f67d-4702-85cd-b46057c8f7d2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:11.119Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f061796a-f67d-4702-85cd-b46057c8f7d2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:14.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2c0f3d4-8d9c-431f-a1d4-a86762f7e209 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2c0f3d4-8d9c-431f-a1d4-a86762f7e209 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2006ee49 2021-05-04T01:00:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2c0f3d4-8d9c-431f-a1d4-a86762f7e209 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2c0f3d4-8d9c-431f-a1d4-a86762f7e209 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:17.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05789e9f-1299-4126-a331-93762e6b7c31 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:17.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05789e9f-1299-4126-a331-93762e6b7c31 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a1ccef4 2021-05-04T01:00:17.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05789e9f-1299-4126-a331-93762e6b7c31 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:17.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05789e9f-1299-4126-a331-93762e6b7c31 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:00:20.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17d4727c-7d47-4bd5-889e-77526e71f816 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:20.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17d4727c-7d47-4bd5-889e-77526e71f816 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64ffb610 2021-05-04T01:00:20.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17d4727c-7d47-4bd5-889e-77526e71f816 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:20.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17d4727c-7d47-4bd5-889e-77526e71f816 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:20.843Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:00:23.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee4b867f-7021-4c5f-b86a-11f133c9a869 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:23.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee4b867f-7021-4c5f-b86a-11f133c9a869 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d999129 2021-05-04T01:00:23.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee4b867f-7021-4c5f-b86a-11f133c9a869 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:23.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee4b867f-7021-4c5f-b86a-11f133c9a869 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:26.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f23d4785-5ab8-419e-97d3-d5a1ca5a6165 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:26.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f23d4785-5ab8-419e-97d3-d5a1ca5a6165 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7afa4e75 2021-05-04T01:00:26.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f23d4785-5ab8-419e-97d3-d5a1ca5a6165 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:26.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f23d4785-5ab8-419e-97d3-d5a1ca5a6165 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:31.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0ebbb60-0372-4d4b-99b9-b30e69bf3b80 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:31.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0ebbb60-0372-4d4b-99b9-b30e69bf3b80 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@462d44b0 2021-05-04T01:00:31.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0ebbb60-0372-4d4b-99b9-b30e69bf3b80 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:31.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0ebbb60-0372-4d4b-99b9-b30e69bf3b80 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:33.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fa70462-37ac-45f2-8774-cc1696c805b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:33.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fa70462-37ac-45f2-8774-cc1696c805b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ffb2553 2021-05-04T01:00:33.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fa70462-37ac-45f2-8774-cc1696c805b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:33.147Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fa70462-37ac-45f2-8774-cc1696c805b4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:00:35.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f34ec4-02f9-4d83-bf20-1da1f211cc35 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:35.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f34ec4-02f9-4d83-bf20-1da1f211cc35 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e4ae75c 2021-05-04T01:00:35.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f34ec4-02f9-4d83-bf20-1da1f211cc35 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:35.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f34ec4-02f9-4d83-bf20-1da1f211cc35 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:37.109Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:00:37.111Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:00:37.111Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:00:38.607Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=41f34ec4-02f9-4d83-bf20-1da1f211cc35 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:00:38.608Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=41f34ec4-02f9-4d83-bf20-1da1f211cc35 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:00:38.609Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=41f34ec4-02f9-4d83-bf20-1da1f211cc35 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:00:38.609Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=41f34ec4-02f9-4d83-bf20-1da1f211cc35 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:00:40.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35980759-a88a-4739-8e9d-38cebff3587a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:40.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35980759-a88a-4739-8e9d-38cebff3587a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@566b9706 2021-05-04T01:00:40.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35980759-a88a-4739-8e9d-38cebff3587a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:40.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35980759-a88a-4739-8e9d-38cebff3587a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:41.757Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=35980759-a88a-4739-8e9d-38cebff3587a ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:00:41.757Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=35980759-a88a-4739-8e9d-38cebff3587a ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:00:42.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2b1703-3c18-4d4e-88ef-e7045c487ae7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:42.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2b1703-3c18-4d4e-88ef-e7045c487ae7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c982208 2021-05-04T01:00:42.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2b1703-3c18-4d4e-88ef-e7045c487ae7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:42.611Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2b1703-3c18-4d4e-88ef-e7045c487ae7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:43.168Z [UEB-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend probably lost connectivity to UEB Cluster. Please check the logs for more information. 2021-05-04T01:00:44.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=636db1aa-7eaf-4df9-892a-3684f02132a4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:44.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=636db1aa-7eaf-4df9-892a-3684f02132a4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55415ac0 2021-05-04T01:00:44.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=636db1aa-7eaf-4df9-892a-3684f02132a4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:44.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=636db1aa-7eaf-4df9-892a-3684f02132a4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:44.709Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=636db1aa-7eaf-4df9-892a-3684f02132a4 ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:00:44.709Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=636db1aa-7eaf-4df9-892a-3684f02132a4 ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:00:47.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5680036-d667-4435-a0c1-c45771ecb8e2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:47.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5680036-d667-4435-a0c1-c45771ecb8e2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f9834c9 2021-05-04T01:00:47.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5680036-d667-4435-a0c1-c45771ecb8e2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:47.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5680036-d667-4435-a0c1-c45771ecb8e2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:50.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37f223be-1f11-470d-ab69-6155202326af PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:50.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37f223be-1f11-470d-ab69-6155202326af PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f5ac42c 2021-05-04T01:00:50.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37f223be-1f11-470d-ab69-6155202326af PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:50.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37f223be-1f11-470d-ab69-6155202326af PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:53.655Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98031f69-0c2a-4ecc-9aeb-78b21a5080e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:53.655Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98031f69-0c2a-4ecc-9aeb-78b21a5080e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@626f25ad 2021-05-04T01:00:53.655Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98031f69-0c2a-4ecc-9aeb-78b21a5080e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:53.655Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98031f69-0c2a-4ecc-9aeb-78b21a5080e5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ee4f8d4-8167-4aae-bb48-20591f025af1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ee4f8d4-8167-4aae-bb48-20591f025af1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d27f47a 2021-05-04T01:00:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ee4f8d4-8167-4aae-bb48-20591f025af1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:56.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ee4f8d4-8167-4aae-bb48-20591f025af1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:00:59.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17518ec6-e7e4-4b68-a98b-a8018bdef7ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:00:59.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17518ec6-e7e4-4b68-a98b-a8018bdef7ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2307b16b 2021-05-04T01:00:59.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17518ec6-e7e4-4b68-a98b-a8018bdef7ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:00:59.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17518ec6-e7e4-4b68-a98b-a8018bdef7ea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:02.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2a0cec-13eb-4e6c-814c-487d1ca6395e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:02.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2a0cec-13eb-4e6c-814c-487d1ca6395e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2af56a83 2021-05-04T01:01:02.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2a0cec-13eb-4e6c-814c-487d1ca6395e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:02.706Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2a0cec-13eb-4e6c-814c-487d1ca6395e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:06.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=face6137-9945-41c2-b814-cbd108bfce4b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:06.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=face6137-9945-41c2-b814-cbd108bfce4b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@312c703d 2021-05-04T01:01:06.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=face6137-9945-41c2-b814-cbd108bfce4b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:06.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=face6137-9945-41c2-b814-cbd108bfce4b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:08.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48a8abd6-2a2b-49c7-a198-0bf313cd16fb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:08.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48a8abd6-2a2b-49c7-a198-0bf313cd16fb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33aad646 2021-05-04T01:01:08.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48a8abd6-2a2b-49c7-a198-0bf313cd16fb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:08.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48a8abd6-2a2b-49c7-a198-0bf313cd16fb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:11.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae805b27-a3db-4ad9-a132-b4c148c1ad61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:11.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae805b27-a3db-4ad9-a132-b4c148c1ad61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d38b5b1 2021-05-04T01:01:11.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae805b27-a3db-4ad9-a132-b4c148c1ad61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:11.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae805b27-a3db-4ad9-a132-b4c148c1ad61 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=406d692c-3d68-4acd-8498-ea005110432f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=406d692c-3d68-4acd-8498-ea005110432f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c9863e7 2021-05-04T01:01:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=406d692c-3d68-4acd-8498-ea005110432f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:14.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=406d692c-3d68-4acd-8498-ea005110432f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:17.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6260a609-1b25-4d0c-a8cd-085b1e6ebb04 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:17.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6260a609-1b25-4d0c-a8cd-085b1e6ebb04 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e6753d1 2021-05-04T01:01:17.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6260a609-1b25-4d0c-a8cd-085b1e6ebb04 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:17.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6260a609-1b25-4d0c-a8cd-085b1e6ebb04 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6cf09bd-c918-4b62-88c7-409e275e2247 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6cf09bd-c918-4b62-88c7-409e275e2247 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b1aa2bd 2021-05-04T01:01:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6cf09bd-c918-4b62-88c7-409e275e2247 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:20.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6cf09bd-c918-4b62-88c7-409e275e2247 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:23.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba4c725f-3dfa-438d-9580-ad8f6a658d7e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:23.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba4c725f-3dfa-438d-9580-ad8f6a658d7e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73670952 2021-05-04T01:01:23.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba4c725f-3dfa-438d-9580-ad8f6a658d7e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:23.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba4c725f-3dfa-438d-9580-ad8f6a658d7e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:26.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b0d5379-2aa2-4439-b71e-fc7806fc3a17 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:26.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b0d5379-2aa2-4439-b71e-fc7806fc3a17 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c0fcb82 2021-05-04T01:01:26.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b0d5379-2aa2-4439-b71e-fc7806fc3a17 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:26.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b0d5379-2aa2-4439-b71e-fc7806fc3a17 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:29.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=357f59f7-6dc3-4a4c-bf04-5aeb8ff5ddcd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:29.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=357f59f7-6dc3-4a4c-bf04-5aeb8ff5ddcd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58b2ea09 2021-05-04T01:01:29.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=357f59f7-6dc3-4a4c-bf04-5aeb8ff5ddcd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:29.499Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=357f59f7-6dc3-4a4c-bf04-5aeb8ff5ddcd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:31.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9a3a4f3-7c4c-4d73-ad21-b439d3efff2f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:31.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9a3a4f3-7c4c-4d73-ad21-b439d3efff2f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1673a4e8 2021-05-04T01:01:31.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9a3a4f3-7c4c-4d73-ad21-b439d3efff2f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:31.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9a3a4f3-7c4c-4d73-ad21-b439d3efff2f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:35.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c1f2299-6f3f-41d3-83b7-7ced3df06b3f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:35.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c1f2299-6f3f-41d3-83b7-7ced3df06b3f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d9502ed 2021-05-04T01:01:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c1f2299-6f3f-41d3-83b7-7ced3df06b3f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:35.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c1f2299-6f3f-41d3-83b7-7ced3df06b3f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:38.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccd7e446-2c26-4e67-b15a-d8402ee17d44 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:38.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccd7e446-2c26-4e67-b15a-d8402ee17d44 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3fa48106 2021-05-04T01:01:38.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccd7e446-2c26-4e67-b15a-d8402ee17d44 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:38.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccd7e446-2c26-4e67-b15a-d8402ee17d44 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:41.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=acf5e0eb-5045-4d45-b75c-5b53ba0f4544 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:41.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=acf5e0eb-5045-4d45-b75c-5b53ba0f4544 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7089c659 2021-05-04T01:01:41.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=acf5e0eb-5045-4d45-b75c-5b53ba0f4544 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:41.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=acf5e0eb-5045-4d45-b75c-5b53ba0f4544 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:44.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0b58aea-7928-4fa3-ba7b-90d056a4474a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:44.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0b58aea-7928-4fa3-ba7b-90d056a4474a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7bd66844 2021-05-04T01:01:44.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0b58aea-7928-4fa3-ba7b-90d056a4474a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:44.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0b58aea-7928-4fa3-ba7b-90d056a4474a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:46.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e03dc1d-38d4-4efc-8bb8-6e8e82608732 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:46.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e03dc1d-38d4-4efc-8bb8-6e8e82608732 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ea19fb9 2021-05-04T01:01:46.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e03dc1d-38d4-4efc-8bb8-6e8e82608732 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:46.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e03dc1d-38d4-4efc-8bb8-6e8e82608732 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:50.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9700d1e9-98ed-4974-be5e-9b34e904b790 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:50.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9700d1e9-98ed-4974-be5e-9b34e904b790 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@151683da 2021-05-04T01:01:50.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9700d1e9-98ed-4974-be5e-9b34e904b790 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:50.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9700d1e9-98ed-4974-be5e-9b34e904b790 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:53.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b7a7273-522b-4d19-90ae-27f7e9281df6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:53.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b7a7273-522b-4d19-90ae-27f7e9281df6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b13342f 2021-05-04T01:01:53.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b7a7273-522b-4d19-90ae-27f7e9281df6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:53.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b7a7273-522b-4d19-90ae-27f7e9281df6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:56.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f515e9cc-24f6-47ee-bbca-0e0f829f24b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:56.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f515e9cc-24f6-47ee-bbca-0e0f829f24b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63cffd41 2021-05-04T01:01:56.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f515e9cc-24f6-47ee-bbca-0e0f829f24b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:56.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f515e9cc-24f6-47ee-bbca-0e0f829f24b4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:01:59.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f38ce40-07a1-48a2-871a-5779a8687f30 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:01:59.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f38ce40-07a1-48a2-871a-5779a8687f30 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6191ead6 2021-05-04T01:01:59.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f38ce40-07a1-48a2-871a-5779a8687f30 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:01:59.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f38ce40-07a1-48a2-871a-5779a8687f30 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:02.598Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfa4f862-b0d1-45e4-a774-437fa761ced0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:02.598Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfa4f862-b0d1-45e4-a774-437fa761ced0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4038ad6d 2021-05-04T01:02:02.598Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfa4f862-b0d1-45e4-a774-437fa761ced0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:02.598Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfa4f862-b0d1-45e4-a774-437fa761ced0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97e6e7f1-1e8b-4efa-998b-7b297e20700f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97e6e7f1-1e8b-4efa-998b-7b297e20700f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72b85b09 2021-05-04T01:02:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97e6e7f1-1e8b-4efa-998b-7b297e20700f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:05.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97e6e7f1-1e8b-4efa-998b-7b297e20700f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:08.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3af1e41-a50b-4ff2-bde3-2a9f99cad3e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:08.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3af1e41-a50b-4ff2-bde3-2a9f99cad3e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e0185ea 2021-05-04T01:02:08.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3af1e41-a50b-4ff2-bde3-2a9f99cad3e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:08.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3af1e41-a50b-4ff2-bde3-2a9f99cad3e1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:11.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ba150f8-156d-478c-b012-0abc028292e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:11.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ba150f8-156d-478c-b012-0abc028292e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a805a6 2021-05-04T01:02:11.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ba150f8-156d-478c-b012-0abc028292e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:11.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ba150f8-156d-478c-b012-0abc028292e5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9692c124-1206-4200-9c24-9fa784c950d0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9692c124-1206-4200-9c24-9fa784c950d0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57e831b 2021-05-04T01:02:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9692c124-1206-4200-9c24-9fa784c950d0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:14.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9692c124-1206-4200-9c24-9fa784c950d0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:17.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6eed08f-689c-4e3f-8a26-a25fe4affbfb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:17.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6eed08f-689c-4e3f-8a26-a25fe4affbfb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35aef0ae 2021-05-04T01:02:17.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6eed08f-689c-4e3f-8a26-a25fe4affbfb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:17.605Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6eed08f-689c-4e3f-8a26-a25fe4affbfb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:20.609Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a6eed08f-689c-4e3f-8a26-a25fe4affbfb ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:02:20.611Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a6eed08f-689c-4e3f-8a26-a25fe4affbfb ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:02:20.611Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a6eed08f-689c-4e3f-8a26-a25fe4affbfb ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:02:20.611Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a6eed08f-689c-4e3f-8a26-a25fe4affbfb ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:02:23.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ba5c93-d766-40f5-b167-ba72f02720b1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:23.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ba5c93-d766-40f5-b167-ba72f02720b1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57621d24 2021-05-04T01:02:23.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ba5c93-d766-40f5-b167-ba72f02720b1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:23.247Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ba5c93-d766-40f5-b167-ba72f02720b1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:25.608Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d8ba5c93-d766-40f5-b167-ba72f02720b1 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:02:25.609Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d8ba5c93-d766-40f5-b167-ba72f02720b1 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:02:27.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf99b712-ee4b-4b32-942d-433ce8188988 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:27.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf99b712-ee4b-4b32-942d-433ce8188988 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66b088ff 2021-05-04T01:02:27.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf99b712-ee4b-4b32-942d-433ce8188988 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:27.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf99b712-ee4b-4b32-942d-433ce8188988 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:02:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:02:30.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0421e30a-6c30-446c-be88-89973ab09768 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:30.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0421e30a-6c30-446c-be88-89973ab09768 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24592f07 2021-05-04T01:02:30.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0421e30a-6c30-446c-be88-89973ab09768 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:30.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0421e30a-6c30-446c-be88-89973ab09768 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:31.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f521368b-7bd0-47a0-b28a-e810054ef1fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:31.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f521368b-7bd0-47a0-b28a-e810054ef1fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63958d95 2021-05-04T01:02:31.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f521368b-7bd0-47a0-b28a-e810054ef1fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:31.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f521368b-7bd0-47a0-b28a-e810054ef1fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:32.304Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f521368b-7bd0-47a0-b28a-e810054ef1fd ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:02:32.305Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f521368b-7bd0-47a0-b28a-e810054ef1fd ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:02:32.306Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:02:32.733Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=032c4d3b-f738-45b3-a0e8-36d74e6e7b14 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:32.733Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=032c4d3b-f738-45b3-a0e8-36d74e6e7b14 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4922f3c3 2021-05-04T01:02:32.733Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=032c4d3b-f738-45b3-a0e8-36d74e6e7b14 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:32.734Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=032c4d3b-f738-45b3-a0e8-36d74e6e7b14 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:35.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d67f0015-8efd-4f8e-a243-0f4e8245de55 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:35.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d67f0015-8efd-4f8e-a243-0f4e8245de55 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b377d68 2021-05-04T01:02:35.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d67f0015-8efd-4f8e-a243-0f4e8245de55 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:35.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d67f0015-8efd-4f8e-a243-0f4e8245de55 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:36.334Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d67f0015-8efd-4f8e-a243-0f4e8245de55 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:02:36.334Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d67f0015-8efd-4f8e-a243-0f4e8245de55 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:02:38.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b5c642e-7800-435c-a709-3641dde602c5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:38.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b5c642e-7800-435c-a709-3641dde602c5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@467b5e1e 2021-05-04T01:02:38.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b5c642e-7800-435c-a709-3641dde602c5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:38.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b5c642e-7800-435c-a709-3641dde602c5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a948ff80-0d44-4f1b-8892-1364b0489e33 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a948ff80-0d44-4f1b-8892-1364b0489e33 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9cd9a61 2021-05-04T01:02:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a948ff80-0d44-4f1b-8892-1364b0489e33 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:41.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a948ff80-0d44-4f1b-8892-1364b0489e33 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:43.515Z [qtp580718781-11] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=128e2466-905a-4799-bd06-48706acf72e3 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T01:02:43.514231Z InvocationID=0b49f789-39cb-4984-995f-e6c9bb06feb1 LogTimestamp=2021-05-04T01:02:43.514552Z Entering 2021-05-04T01:02:43.515Z [qtp580718781-11] INFO o.o.sdc.be.filters.BeServletFilter RequestID=128e2466-905a-4799-bd06-48706acf72e3 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/artifactTypes ServiceInstanceID=null EntryTimestamp=2021-05-04T01:02:43.514231Z InvocationID=0b49f789-39cb-4984-995f-e6c9bb06feb1 LogTimestamp=2021-05-04T01:02:43.514552Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 GET /sdc/v1/artifactTypes HTTP/1.1 2021-05-04T01:02:43.533Z [qtp580718781-11] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=200 RequestID=null ElapsedTime=16 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/artifactTypes EntryTimestamp=2021-05-04T01:02:43.516Z InvocationID=679afc4f-efb9-40dc-b8f1-37c8550d9fa1 LogTimestamp=2021-05-04T01:02:43.532909Z ResponseDesc=OK Exiting. 2021-05-04T01:02:43.533Z [qtp580718781-11] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null GET /sdc/v1/artifactTypes HTTP/1.1 SC="200" 2021-05-04T01:02:44.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cefbc546-e6bc-44e0-98ea-e532d615816d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:44.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cefbc546-e6bc-44e0-98ea-e532d615816d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@572fdc70 2021-05-04T01:02:44.352Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cefbc546-e6bc-44e0-98ea-e532d615816d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:44.352Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cefbc546-e6bc-44e0-98ea-e532d615816d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:46.484Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:46.489Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:46.646Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:46.647Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:47.111Z [qtp580718781-19] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=3d363f3b-ae25-4acf-b26c-881c6e313e36 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T01:02:47.111355Z InvocationID=db617c39-cfd3-4ada-acd2-2b33a33580ff LogTimestamp=2021-05-04T01:02:47.111423Z Entering 2021-05-04T01:02:47.111Z [qtp580718781-19] INFO o.o.sdc.be.filters.BeServletFilter RequestID=3d363f3b-ae25-4acf-b26c-881c6e313e36 InstanceID=f88c5bd5-7859-4eb0-92b4-cf406fcb95b6 ElapsedTime=0 Severity=0 PartnerName=dcae auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh ClientIPAddress=10.242.8.151 StatusCode=INPROGRESS userId=null RemoteHost=10.242.8.151 ServerIPAddress=10.242.235.154 ServiceName=v1/registerForDistribution ServiceInstanceID=null EntryTimestamp=2021-05-04T01:02:47.111355Z InvocationID=db617c39-cfd3-4ada-acd2-2b33a33580ff LogTimestamp=2021-05-04T01:02:47.111423Z localAddr=10.242.235.154 remoteAddr=10.242.8.151 POST /sdc/v1/registerForDistribution HTTP/1.1 2021-05-04T01:02:47.117Z [qtp580718781-19] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T01:02:47.111Z InvocationID=082ea3d9-d240-46b1-9d2a-2563a5ce598f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/Exte8PnzSzMSQ9LB will send credentials over a clear channel. 2021-05-04T01:02:47.117Z [qtp580718781-19] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T01:02:47.111Z InvocationID=082ea3d9-d240-46b1-9d2a-2563a5ce598f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-STATUS-TOPIC-AUTO/producers/Exte8PnzSzMSQ9LB (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:02:47.445Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f270567-c8e8-4a6e-a70a-6a759bc0fd24 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:47.445Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f270567-c8e8-4a6e-a70a-6a759bc0fd24 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47e327c0 2021-05-04T01:02:47.445Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f270567-c8e8-4a6e-a70a-6a759bc0fd24 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:47.445Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f270567-c8e8-4a6e-a70a-6a759bc0fd24 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:49.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:49.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:49.976Z [qtp580718781-19] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T01:02:47.111Z InvocationID=082ea3d9-d240-46b1-9d2a-2563a5ce598f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.042Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.043Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.051Z [qtp580718781-19] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "Exte8PnzSzMSQ9LB" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T01:02:47.111Z InvocationID=082ea3d9-d240-46b1-9d2a-2563a5ce598f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/Exte8PnzSzMSQ9LB will send credentials over a clear channel. 2021-05-04T01:02:50.052Z [qtp580718781-19] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "Exte8PnzSzMSQ9LB" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T01:02:47.111Z InvocationID=082ea3d9-d240-46b1-9d2a-2563a5ce598f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh PUT http://message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/consumers/Exte8PnzSzMSQ9LB (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:02:50.057Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.058Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.073Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.075Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.086Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.087Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.100Z [qtp580718781-19] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-STATUS-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "Exte8PnzSzMSQ9LB" STATUS = "200" RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T01:02:47.111Z InvocationID=082ea3d9-d240-46b1-9d2a-2563a5ce598f PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.110Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.111Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.135Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.136Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.145Z [qtp580718781-19] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "DRegister" CONSUMER_ID = "ves-openapi-manager" API_KEY = "Exte8PnzSzMSQ9LB" D_ENV = "AUTO" STATUS = "200" DESC = "" DNOTIF_TOPIC = "SDC-DISTR-NOTIF-TOPIC-AUTO" DSTATUS_TOPIC = "SDC-DISTR-STATUS-TOPIC-AUTO" ResponseCode=200 RequestID=null ElapsedTime=3034 PartnerName=Apache-HttpClient/4.5.13 (Java/11.0.9.1) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh StatusCode=COMPLETE ServerIPAddress=10.242.235.154 ServiceName=/v1/registerForDistribution EntryTimestamp=2021-05-04T01:02:47.111Z InvocationID=082ea3d9-d240-46b1-9d2a-2563a5ce598f LogTimestamp=2021-05-04T01:02:50.145750Z ResponseDesc=OK Exiting. 2021-05-04T01:02:50.146Z [qtp580718781-19] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc/v1/registerForDistribution HTTP/1.1 SC="200" 2021-05-04T01:02:50.151Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.152Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.162Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.163Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.188Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.189Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.198Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.199Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.204Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.205Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.213Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.214Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.224Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.225Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:02:50.259Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:02:50.259Z [UEB-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend connection recovery to UEB Cluster. 2021-05-04T01:02:50.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6832ab4-2d80-4767-9a03-5dbfe2202271 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:50.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6832ab4-2d80-4767-9a03-5dbfe2202271 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b741a8c 2021-05-04T01:02:50.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6832ab4-2d80-4767-9a03-5dbfe2202271 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:50.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6832ab4-2d80-4767-9a03-5dbfe2202271 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=959c409e-94fb-4a04-a1f5-3139f36e57e4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=959c409e-94fb-4a04-a1f5-3139f36e57e4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e8f8d11 2021-05-04T01:02:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=959c409e-94fb-4a04-a1f5-3139f36e57e4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=959c409e-94fb-4a04-a1f5-3139f36e57e4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:53.704Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=959c409e-94fb-4a04-a1f5-3139f36e57e4 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:02:53.710Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=959c409e-94fb-4a04-a1f5-3139f36e57e4 ErrorCategory=INFO ServiceName=BE Component DE state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:02:56.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6485e6c-33da-4f38-b762-fc25ad42fe6b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6485e6c-33da-4f38-b762-fc25ad42fe6b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17ea1aec 2021-05-04T01:02:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6485e6c-33da-4f38-b762-fc25ad42fe6b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6485e6c-33da-4f38-b762-fc25ad42fe6b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:02:59.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c2503bb-8e2a-430a-864b-a7fedab66030 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:02:59.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c2503bb-8e2a-430a-864b-a7fedab66030 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49e9f2f6 2021-05-04T01:02:59.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c2503bb-8e2a-430a-864b-a7fedab66030 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:02:59.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c2503bb-8e2a-430a-864b-a7fedab66030 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:02.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=862bc1bf-1c16-47e8-a957-ae510e0df515 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=862bc1bf-1c16-47e8-a957-ae510e0df515 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3232213d 2021-05-04T01:03:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=862bc1bf-1c16-47e8-a957-ae510e0df515 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=862bc1bf-1c16-47e8-a957-ae510e0df515 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:03.808Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T01:03:03.809Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:03:03.810Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:03:03.810Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:03:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:03:05.220Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:03:05.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=235f59b1-eda9-4801-a918-183d6eb82277 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:05.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=235f59b1-eda9-4801-a918-183d6eb82277 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7097c56f 2021-05-04T01:03:05.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=235f59b1-eda9-4801-a918-183d6eb82277 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:05.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=235f59b1-eda9-4801-a918-183d6eb82277 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:08.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2138df3f-ea97-41d6-b96f-0a0725c03560 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:08.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2138df3f-ea97-41d6-b96f-0a0725c03560 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@376d8ceb 2021-05-04T01:03:08.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2138df3f-ea97-41d6-b96f-0a0725c03560 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:08.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2138df3f-ea97-41d6-b96f-0a0725c03560 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:11.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2496bf21-0841-40a2-8fee-df1ec9601c57 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:11.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2496bf21-0841-40a2-8fee-df1ec9601c57 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3115a6b2 2021-05-04T01:03:11.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2496bf21-0841-40a2-8fee-df1ec9601c57 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:11.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2496bf21-0841-40a2-8fee-df1ec9601c57 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:14.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=173a96b5-04b6-4120-9da4-ba40e4087a4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:14.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=173a96b5-04b6-4120-9da4-ba40e4087a4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@128d4264 2021-05-04T01:03:14.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=173a96b5-04b6-4120-9da4-ba40e4087a4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:14.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=173a96b5-04b6-4120-9da4-ba40e4087a4a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:17.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=934d50d4-0f8f-412d-b3de-2bdd68b9a95c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:17.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=934d50d4-0f8f-412d-b3de-2bdd68b9a95c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5cff41e7 2021-05-04T01:03:17.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=934d50d4-0f8f-412d-b3de-2bdd68b9a95c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:17.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=934d50d4-0f8f-412d-b3de-2bdd68b9a95c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:19.274Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T01:03:19.274Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:03:19.275Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:03:19.275Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:03:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:03:20.347Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:03:20.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ca2369b-3b0b-4954-a7fc-fe6fe82f9c83 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:20.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ca2369b-3b0b-4954-a7fc-fe6fe82f9c83 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72e8447 2021-05-04T01:03:20.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ca2369b-3b0b-4954-a7fc-fe6fe82f9c83 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:20.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ca2369b-3b0b-4954-a7fc-fe6fe82f9c83 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e950699-a685-4011-bd82-00a28baa8366 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e950699-a685-4011-bd82-00a28baa8366 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b129226 2021-05-04T01:03:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e950699-a685-4011-bd82-00a28baa8366 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e950699-a685-4011-bd82-00a28baa8366 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:26.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e51739fd-8801-40cc-9c9f-b9c456d7d05a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:26.352Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e51739fd-8801-40cc-9c9f-b9c456d7d05a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63f46d42 2021-05-04T01:03:26.352Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e51739fd-8801-40cc-9c9f-b9c456d7d05a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:26.352Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e51739fd-8801-40cc-9c9f-b9c456d7d05a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:29.209Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa698bed-27f5-4fc4-8539-17f5aaf3e80d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:29.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa698bed-27f5-4fc4-8539-17f5aaf3e80d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4522e8a2 2021-05-04T01:03:29.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa698bed-27f5-4fc4-8539-17f5aaf3e80d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:29.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa698bed-27f5-4fc4-8539-17f5aaf3e80d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:32.154Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e830197b-90c2-4de9-bf0f-17f63e4e75ed PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e830197b-90c2-4de9-bf0f-17f63e4e75ed PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@365d50fd 2021-05-04T01:03:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e830197b-90c2-4de9-bf0f-17f63e4e75ed PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e830197b-90c2-4de9-bf0f-17f63e4e75ed PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:34.227Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T01:03:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:03:35.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:03:35.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=59d7ffef-e28b-4e0c-9276-fdbf00b6c7dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:35.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=59d7ffef-e28b-4e0c-9276-fdbf00b6c7dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@395177b2 2021-05-04T01:03:35.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=59d7ffef-e28b-4e0c-9276-fdbf00b6c7dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:35.307Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=59d7ffef-e28b-4e0c-9276-fdbf00b6c7dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:03:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:03:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:03:38.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea4f9d4d-a2e9-4908-b05b-067c5e8489da PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:38.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea4f9d4d-a2e9-4908-b05b-067c5e8489da PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@790f83a7 2021-05-04T01:03:38.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea4f9d4d-a2e9-4908-b05b-067c5e8489da PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:38.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea4f9d4d-a2e9-4908-b05b-067c5e8489da PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:41.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9402c3f6-f810-4c6c-bb05-1871ed513341 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:41.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9402c3f6-f810-4c6c-bb05-1871ed513341 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b3a13dc 2021-05-04T01:03:41.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9402c3f6-f810-4c6c-bb05-1871ed513341 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:41.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9402c3f6-f810-4c6c-bb05-1871ed513341 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:44.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f9725f1-6944-4ed3-a000-91c0cca5e193 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:44.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f9725f1-6944-4ed3-a000-91c0cca5e193 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@773c74fe 2021-05-04T01:03:44.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f9725f1-6944-4ed3-a000-91c0cca5e193 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:44.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f9725f1-6944-4ed3-a000-91c0cca5e193 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:46.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae70e2fc-a748-4c84-8b84-fcd6703b637f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:46.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae70e2fc-a748-4c84-8b84-fcd6703b637f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45e2614c 2021-05-04T01:03:46.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae70e2fc-a748-4c84-8b84-fcd6703b637f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:46.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae70e2fc-a748-4c84-8b84-fcd6703b637f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:03:50.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:03:50.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d447f3b9-c3e7-4754-80c1-9faad37ba382 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:50.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d447f3b9-c3e7-4754-80c1-9faad37ba382 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40f5d0ec 2021-05-04T01:03:50.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d447f3b9-c3e7-4754-80c1-9faad37ba382 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:50.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d447f3b9-c3e7-4754-80c1-9faad37ba382 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:52.072Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T01:03:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7f2e04f-3a1e-4236-9684-a771b70d484b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7f2e04f-3a1e-4236-9684-a771b70d484b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3baa9fdd 2021-05-04T01:03:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7f2e04f-3a1e-4236-9684-a771b70d484b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7f2e04f-3a1e-4236-9684-a771b70d484b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:56.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44fe730c-ff1a-4688-9ad6-d3d0b51331f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:56.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44fe730c-ff1a-4688-9ad6-d3d0b51331f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f2587d2 2021-05-04T01:03:56.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44fe730c-ff1a-4688-9ad6-d3d0b51331f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:56.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44fe730c-ff1a-4688-9ad6-d3d0b51331f6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:03:59.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb216a74-39d3-4d44-8bfe-d0c4b3f6e269 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:03:59.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb216a74-39d3-4d44-8bfe-d0c4b3f6e269 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@791b6b7b 2021-05-04T01:03:59.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb216a74-39d3-4d44-8bfe-d0c4b3f6e269 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:03:59.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb216a74-39d3-4d44-8bfe-d0c4b3f6e269 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc045722-08c7-4066-a187-707acae5e2ab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc045722-08c7-4066-a187-707acae5e2ab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1707b604 2021-05-04T01:04:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc045722-08c7-4066-a187-707acae5e2ab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:02.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc045722-08c7-4066-a187-707acae5e2ab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:04:05.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:04:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0952e9a3-bac9-4a88-947e-03f244434874 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:05.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0952e9a3-bac9-4a88-947e-03f244434874 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@783c874e 2021-05-04T01:04:05.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0952e9a3-bac9-4a88-947e-03f244434874 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:05.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0952e9a3-bac9-4a88-947e-03f244434874 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:08.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1e44bc3-24ec-40cc-a364-34e573a44ddc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:08.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1e44bc3-24ec-40cc-a364-34e573a44ddc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b24948c 2021-05-04T01:04:08.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1e44bc3-24ec-40cc-a364-34e573a44ddc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:08.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1e44bc3-24ec-40cc-a364-34e573a44ddc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec38763a-ff68-4033-a944-ef8a719446d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec38763a-ff68-4033-a944-ef8a719446d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59467820 2021-05-04T01:04:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec38763a-ff68-4033-a944-ef8a719446d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:11.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec38763a-ff68-4033-a944-ef8a719446d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55a25fc9-6a48-4076-b76d-1132c48ae0cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55a25fc9-6a48-4076-b76d-1132c48ae0cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@735b429a 2021-05-04T01:04:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55a25fc9-6a48-4076-b76d-1132c48ae0cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:14.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55a25fc9-6a48-4076-b76d-1132c48ae0cd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:17.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d59232d-ab6c-408d-932e-9e821d6212a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:17.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d59232d-ab6c-408d-932e-9e821d6212a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f41bd48 2021-05-04T01:04:17.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d59232d-ab6c-408d-932e-9e821d6212a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:17.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d59232d-ab6c-408d-932e-9e821d6212a0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:20.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=588d0bb3-69c9-4de1-a2d9-e03fd0689b4d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:20.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=588d0bb3-69c9-4de1-a2d9-e03fd0689b4d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@301ec4d8 2021-05-04T01:04:20.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=588d0bb3-69c9-4de1-a2d9-e03fd0689b4d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:20.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=588d0bb3-69c9-4de1-a2d9-e03fd0689b4d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:04:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:04:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc905e26-1306-4bd6-a021-0476526acbfb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc905e26-1306-4bd6-a021-0476526acbfb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a732d83 2021-05-04T01:04:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc905e26-1306-4bd6-a021-0476526acbfb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc905e26-1306-4bd6-a021-0476526acbfb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:26.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8138637b-a4b1-4a9e-b3e9-1f8825530bae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:26.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8138637b-a4b1-4a9e-b3e9-1f8825530bae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7395fbc5 2021-05-04T01:04:26.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8138637b-a4b1-4a9e-b3e9-1f8825530bae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:26.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8138637b-a4b1-4a9e-b3e9-1f8825530bae PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:29.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abd3ea46-4c7c-4c90-9293-52d7b562fa16 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:29.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abd3ea46-4c7c-4c90-9293-52d7b562fa16 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@115cb302 2021-05-04T01:04:29.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abd3ea46-4c7c-4c90-9293-52d7b562fa16 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:29.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abd3ea46-4c7c-4c90-9293-52d7b562fa16 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:32.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e237f398-bc27-41b1-8f30-e08b35e500ee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:32.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e237f398-bc27-41b1-8f30-e08b35e500ee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57229df5 2021-05-04T01:04:32.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e237f398-bc27-41b1-8f30-e08b35e500ee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:32.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e237f398-bc27-41b1-8f30-e08b35e500ee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:04:35.188Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:04:35.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654c1e35-25b1-48ae-9744-b68aaa72fa60 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:35.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654c1e35-25b1-48ae-9744-b68aaa72fa60 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14feada7 2021-05-04T01:04:35.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654c1e35-25b1-48ae-9744-b68aaa72fa60 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:35.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654c1e35-25b1-48ae-9744-b68aaa72fa60 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:37.109Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:04:37.110Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:04:37.110Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:04:38.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5516912d-fec9-450d-a447-90b6a0c45e44 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:38.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5516912d-fec9-450d-a447-90b6a0c45e44 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48e9372e 2021-05-04T01:04:38.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5516912d-fec9-450d-a447-90b6a0c45e44 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:38.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5516912d-fec9-450d-a447-90b6a0c45e44 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:41.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd6d8d29-d23d-4915-baee-60fead8a469a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:41.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd6d8d29-d23d-4915-baee-60fead8a469a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c595992 2021-05-04T01:04:41.108Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd6d8d29-d23d-4915-baee-60fead8a469a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:41.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd6d8d29-d23d-4915-baee-60fead8a469a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:44.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bff6bbd4-1cbc-43dc-ad08-9ac36fcda8d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:44.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bff6bbd4-1cbc-43dc-ad08-9ac36fcda8d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f57a6d9 2021-05-04T01:04:44.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bff6bbd4-1cbc-43dc-ad08-9ac36fcda8d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:44.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bff6bbd4-1cbc-43dc-ad08-9ac36fcda8d9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:46.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37b3595d-1565-45fc-8a2f-c28dadcc2016 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:46.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37b3595d-1565-45fc-8a2f-c28dadcc2016 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e25bf01 2021-05-04T01:04:46.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37b3595d-1565-45fc-8a2f-c28dadcc2016 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:46.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37b3595d-1565-45fc-8a2f-c28dadcc2016 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:50.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:04:50.184Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:04:50.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ab7f0d-53f8-4019-8288-3ca64631f2e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:50.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ab7f0d-53f8-4019-8288-3ca64631f2e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f74abac 2021-05-04T01:04:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ab7f0d-53f8-4019-8288-3ca64631f2e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ab7f0d-53f8-4019-8288-3ca64631f2e7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:52.069Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T01:04:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7d841c9-b9d0-450b-bffe-0e2d73b1fb7d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7d841c9-b9d0-450b-bffe-0e2d73b1fb7d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6917f419 2021-05-04T01:04:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7d841c9-b9d0-450b-bffe-0e2d73b1fb7d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:53.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7d841c9-b9d0-450b-bffe-0e2d73b1fb7d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c91b2444-741e-49f6-80e5-a167e010ff38 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c91b2444-741e-49f6-80e5-a167e010ff38 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b34e68c 2021-05-04T01:04:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c91b2444-741e-49f6-80e5-a167e010ff38 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:56.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c91b2444-741e-49f6-80e5-a167e010ff38 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:04:59.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f8d1a7-0ec2-41e4-aae6-e18c3c3746f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:04:59.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f8d1a7-0ec2-41e4-aae6-e18c3c3746f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18da995c 2021-05-04T01:04:59.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f8d1a7-0ec2-41e4-aae6-e18c3c3746f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:04:59.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f8d1a7-0ec2-41e4-aae6-e18c3c3746f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:02.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d029fa4-a193-4e7a-b78b-8080b35670e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:02.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d029fa4-a193-4e7a-b78b-8080b35670e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22496820 2021-05-04T01:05:02.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d029fa4-a193-4e7a-b78b-8080b35670e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:02.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d029fa4-a193-4e7a-b78b-8080b35670e7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:05:05.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:05:05.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1b36b888-4087-467a-ba17-405570c297c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:05.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1b36b888-4087-467a-ba17-405570c297c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74a8e74c 2021-05-04T01:05:05.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1b36b888-4087-467a-ba17-405570c297c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:05.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1b36b888-4087-467a-ba17-405570c297c1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d1d01e7-e780-4fcf-8cbe-9b56d0a5573c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d1d01e7-e780-4fcf-8cbe-9b56d0a5573c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78a46fd3 2021-05-04T01:05:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d1d01e7-e780-4fcf-8cbe-9b56d0a5573c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:08.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d1d01e7-e780-4fcf-8cbe-9b56d0a5573c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:11.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7798c1e-1c7c-480c-91a3-a473fdd6cf4c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:11.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7798c1e-1c7c-480c-91a3-a473fdd6cf4c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34f809e2 2021-05-04T01:05:11.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7798c1e-1c7c-480c-91a3-a473fdd6cf4c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:11.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7798c1e-1c7c-480c-91a3-a473fdd6cf4c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:14.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2532bc6b-74d5-417b-a249-a4ea986f5153 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:14.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2532bc6b-74d5-417b-a249-a4ea986f5153 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d996f35 2021-05-04T01:05:14.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2532bc6b-74d5-417b-a249-a4ea986f5153 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:14.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2532bc6b-74d5-417b-a249-a4ea986f5153 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:17.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9688aad0-d6db-4add-a631-3b6e731c34c5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:17.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9688aad0-d6db-4add-a631-3b6e731c34c5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e38714d 2021-05-04T01:05:17.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9688aad0-d6db-4add-a631-3b6e731c34c5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:17.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9688aad0-d6db-4add-a631-3b6e731c34c5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:05:20.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:05:20.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4404956-73a9-4f2e-87f8-cf37c21c0026 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:20.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4404956-73a9-4f2e-87f8-cf37c21c0026 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35b951ae 2021-05-04T01:05:20.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4404956-73a9-4f2e-87f8-cf37c21c0026 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:20.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4404956-73a9-4f2e-87f8-cf37c21c0026 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=094ec22c-152a-437b-a437-058f042ed22f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=094ec22c-152a-437b-a437-058f042ed22f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41faf9e2 2021-05-04T01:05:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=094ec22c-152a-437b-a437-058f042ed22f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=094ec22c-152a-437b-a437-058f042ed22f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:26.376Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=811f35f2-e626-4889-b8c5-dc08b51e8aaa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:26.377Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=811f35f2-e626-4889-b8c5-dc08b51e8aaa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@578cc62f 2021-05-04T01:05:26.377Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=811f35f2-e626-4889-b8c5-dc08b51e8aaa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:26.377Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=811f35f2-e626-4889-b8c5-dc08b51e8aaa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:29.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c2d9345-29c8-4dd6-9626-818fc333c418 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:29.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c2d9345-29c8-4dd6-9626-818fc333c418 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a93678a 2021-05-04T01:05:29.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c2d9345-29c8-4dd6-9626-818fc333c418 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:29.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c2d9345-29c8-4dd6-9626-818fc333c418 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b30cf8ea-5d18-42de-a3cb-3925679da1b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b30cf8ea-5d18-42de-a3cb-3925679da1b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5586d57 2021-05-04T01:05:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b30cf8ea-5d18-42de-a3cb-3925679da1b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b30cf8ea-5d18-42de-a3cb-3925679da1b8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:05:35.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:05:35.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39794c2-9c2e-4182-a379-1b7cd50badb5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:35.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39794c2-9c2e-4182-a379-1b7cd50badb5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@137bb1f7 2021-05-04T01:05:35.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39794c2-9c2e-4182-a379-1b7cd50badb5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:35.509Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39794c2-9c2e-4182-a379-1b7cd50badb5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:05:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:05:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:05:38.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4eaca68-3c49-4b83-a309-09f4d05148fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:38.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4eaca68-3c49-4b83-a309-09f4d05148fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22f7b009 2021-05-04T01:05:38.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4eaca68-3c49-4b83-a309-09f4d05148fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:38.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4eaca68-3c49-4b83-a309-09f4d05148fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:41.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdba81eb-4085-43bc-91b1-e46af498b3d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:41.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdba81eb-4085-43bc-91b1-e46af498b3d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43392cea 2021-05-04T01:05:41.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdba81eb-4085-43bc-91b1-e46af498b3d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:41.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdba81eb-4085-43bc-91b1-e46af498b3d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973cfdb3-3a43-42b1-8004-c530666732a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973cfdb3-3a43-42b1-8004-c530666732a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5cdc30f 2021-05-04T01:05:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973cfdb3-3a43-42b1-8004-c530666732a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973cfdb3-3a43-42b1-8004-c530666732a3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:47.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab7f14b5-41dc-4807-87e8-1a606e969521 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:47.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab7f14b5-41dc-4807-87e8-1a606e969521 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4bf72e48 2021-05-04T01:05:47.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab7f14b5-41dc-4807-87e8-1a606e969521 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:47.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab7f14b5-41dc-4807-87e8-1a606e969521 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:05:50.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:05:50.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97508405-5993-4217-a63b-8b92a68d87f0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:50.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97508405-5993-4217-a63b-8b92a68d87f0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@541fa8c7 2021-05-04T01:05:50.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97508405-5993-4217-a63b-8b92a68d87f0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:50.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97508405-5993-4217-a63b-8b92a68d87f0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:52.064Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T01:05:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=709be2c7-dff1-4fe4-81ac-1b15a6389e86 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=709be2c7-dff1-4fe4-81ac-1b15a6389e86 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ecb90d2 2021-05-04T01:05:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=709be2c7-dff1-4fe4-81ac-1b15a6389e86 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:53.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=709be2c7-dff1-4fe4-81ac-1b15a6389e86 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:56.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11866bbe-b79c-4a98-a915-03613a9fd180 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:56.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11866bbe-b79c-4a98-a915-03613a9fd180 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35f5f5bf 2021-05-04T01:05:56.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11866bbe-b79c-4a98-a915-03613a9fd180 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:56.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11866bbe-b79c-4a98-a915-03613a9fd180 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:05:59.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c5b8736-f98b-419e-96d4-e21cbb4c7332 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:05:59.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c5b8736-f98b-419e-96d4-e21cbb4c7332 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47820827 2021-05-04T01:05:59.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c5b8736-f98b-419e-96d4-e21cbb4c7332 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:05:59.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c5b8736-f98b-419e-96d4-e21cbb4c7332 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:02.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=889b10b9-2005-4c9a-acca-f62d86910c88 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:02.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=889b10b9-2005-4c9a-acca-f62d86910c88 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ef2c980 2021-05-04T01:06:02.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=889b10b9-2005-4c9a-acca-f62d86910c88 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:02.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=889b10b9-2005-4c9a-acca-f62d86910c88 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:05.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:06:05.187Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:06:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46002d9d-0dcb-47e7-8477-56e131e94be6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46002d9d-0dcb-47e7-8477-56e131e94be6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6cbd898c 2021-05-04T01:06:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46002d9d-0dcb-47e7-8477-56e131e94be6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:05.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46002d9d-0dcb-47e7-8477-56e131e94be6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb59eb12-28e6-4771-99d6-1f528c9ed42a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb59eb12-28e6-4771-99d6-1f528c9ed42a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76ffd1a9 2021-05-04T01:06:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb59eb12-28e6-4771-99d6-1f528c9ed42a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:08.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb59eb12-28e6-4771-99d6-1f528c9ed42a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:11.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1227b3a1-9276-4f7d-8c9a-38b0567dfb84 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:11.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1227b3a1-9276-4f7d-8c9a-38b0567dfb84 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@216b2c61 2021-05-04T01:06:11.427Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1227b3a1-9276-4f7d-8c9a-38b0567dfb84 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:11.427Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1227b3a1-9276-4f7d-8c9a-38b0567dfb84 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:14.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=866bea73-b0dd-4b09-b9c7-614f90bd6c84 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:14.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=866bea73-b0dd-4b09-b9c7-614f90bd6c84 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@592d5f05 2021-05-04T01:06:14.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=866bea73-b0dd-4b09-b9c7-614f90bd6c84 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:14.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=866bea73-b0dd-4b09-b9c7-614f90bd6c84 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:17.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c24ce13-5e7b-456a-a282-1410839c9190 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:17.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c24ce13-5e7b-456a-a282-1410839c9190 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@595c5c45 2021-05-04T01:06:17.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c24ce13-5e7b-456a-a282-1410839c9190 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:17.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c24ce13-5e7b-456a-a282-1410839c9190 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:20.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c6b5f5e-ef57-46f9-b7b3-24324b82b8ec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:20.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c6b5f5e-ef57-46f9-b7b3-24324b82b8ec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51a6bc5e 2021-05-04T01:06:20.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c6b5f5e-ef57-46f9-b7b3-24324b82b8ec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:20.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c6b5f5e-ef57-46f9-b7b3-24324b82b8ec PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:06:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:06:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32c027e7-69cb-4ca7-8442-ab294dc550cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32c027e7-69cb-4ca7-8442-ab294dc550cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29d68a2f 2021-05-04T01:06:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32c027e7-69cb-4ca7-8442-ab294dc550cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32c027e7-69cb-4ca7-8442-ab294dc550cf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:26.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cfbfb09-8ca0-43c2-90ce-80c7a923b55c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:26.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cfbfb09-8ca0-43c2-90ce-80c7a923b55c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ab03d67 2021-05-04T01:06:26.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cfbfb09-8ca0-43c2-90ce-80c7a923b55c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:26.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cfbfb09-8ca0-43c2-90ce-80c7a923b55c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:29.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0242f9a7-5336-405d-96da-541c83d59485 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:29.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0242f9a7-5336-405d-96da-541c83d59485 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21b5f400 2021-05-04T01:06:29.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0242f9a7-5336-405d-96da-541c83d59485 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:29.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0242f9a7-5336-405d-96da-541c83d59485 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:32.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cb0dab4-bbc2-49f8-821e-8ad21c37736d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:32.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cb0dab4-bbc2-49f8-821e-8ad21c37736d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4515e824 2021-05-04T01:06:32.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cb0dab4-bbc2-49f8-821e-8ad21c37736d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:32.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cb0dab4-bbc2-49f8-821e-8ad21c37736d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:06:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:06:35.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0becab5-5c59-4f43-bc55-33d1734d328a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:35.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0becab5-5c59-4f43-bc55-33d1734d328a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@588f955b 2021-05-04T01:06:35.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0becab5-5c59-4f43-bc55-33d1734d328a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:35.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0becab5-5c59-4f43-bc55-33d1734d328a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:06:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:06:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:06:38.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08c88dc8-8437-43cc-9a29-d6e8fad50e90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:38.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08c88dc8-8437-43cc-9a29-d6e8fad50e90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@434fc028 2021-05-04T01:06:38.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08c88dc8-8437-43cc-9a29-d6e8fad50e90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:38.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08c88dc8-8437-43cc-9a29-d6e8fad50e90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:41.113Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd75078d-92cc-4ea9-8d0e-0282f2424f49 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:41.113Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd75078d-92cc-4ea9-8d0e-0282f2424f49 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d5439b5 2021-05-04T01:06:41.113Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd75078d-92cc-4ea9-8d0e-0282f2424f49 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:41.113Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd75078d-92cc-4ea9-8d0e-0282f2424f49 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:44.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4b6383f-8478-45e7-bb19-d3b156fea88c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:44.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4b6383f-8478-45e7-bb19-d3b156fea88c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27eabcf3 2021-05-04T01:06:44.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4b6383f-8478-45e7-bb19-d3b156fea88c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:44.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4b6383f-8478-45e7-bb19-d3b156fea88c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:47.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=718b7627-3775-4fe2-9d57-a3dd4c54a6ee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:47.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=718b7627-3775-4fe2-9d57-a3dd4c54a6ee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ebd018 2021-05-04T01:06:47.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=718b7627-3775-4fe2-9d57-a3dd4c54a6ee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:47.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=718b7627-3775-4fe2-9d57-a3dd4c54a6ee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:50.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0531a79-e9b4-4274-9f19-b790ce5e5e82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:50.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0531a79-e9b4-4274-9f19-b790ce5e5e82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e181ecd 2021-05-04T01:06:50.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0531a79-e9b4-4274-9f19-b790ce5e5e82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:50.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0531a79-e9b4-4274-9f19-b790ce5e5e82 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:06:50.173Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:06:52.045Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T01:06:53.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4474571-1889-4917-b727-dc835c1450bf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:53.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4474571-1889-4917-b727-dc835c1450bf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52ed012a 2021-05-04T01:06:53.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4474571-1889-4917-b727-dc835c1450bf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:53.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4474571-1889-4917-b727-dc835c1450bf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:56.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a69b5fd-2426-4a3b-bcd3-a982e4113881 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:56.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a69b5fd-2426-4a3b-bcd3-a982e4113881 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@209232a0 2021-05-04T01:06:56.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a69b5fd-2426-4a3b-bcd3-a982e4113881 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:56.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a69b5fd-2426-4a3b-bcd3-a982e4113881 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:06:59.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=172bfd7a-8eea-4eef-a1b4-fe996aef0c07 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:06:59.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=172bfd7a-8eea-4eef-a1b4-fe996aef0c07 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7da45dba 2021-05-04T01:06:59.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=172bfd7a-8eea-4eef-a1b4-fe996aef0c07 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:06:59.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=172bfd7a-8eea-4eef-a1b4-fe996aef0c07 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:02.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52a8c04f-f2db-438b-902b-f7f948c40c82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:02.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52a8c04f-f2db-438b-902b-f7f948c40c82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@334dab5 2021-05-04T01:07:02.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52a8c04f-f2db-438b-902b-f7f948c40c82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:02.422Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52a8c04f-f2db-438b-902b-f7f948c40c82 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:07:05.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:07:05.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a372f813-4574-4a28-912d-9e7dceaab978 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:05.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a372f813-4574-4a28-912d-9e7dceaab978 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@396a7fa8 2021-05-04T01:07:05.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a372f813-4574-4a28-912d-9e7dceaab978 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:05.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a372f813-4574-4a28-912d-9e7dceaab978 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:08.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5767b779-352d-4e88-834f-42d7945da314 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:08.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5767b779-352d-4e88-834f-42d7945da314 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@85c4d17 2021-05-04T01:07:08.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5767b779-352d-4e88-834f-42d7945da314 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:08.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5767b779-352d-4e88-834f-42d7945da314 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:11.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e8d4b61-5686-41f6-b53f-6f0c822316fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:11.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e8d4b61-5686-41f6-b53f-6f0c822316fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57ca882b 2021-05-04T01:07:11.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e8d4b61-5686-41f6-b53f-6f0c822316fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:11.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e8d4b61-5686-41f6-b53f-6f0c822316fa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:14.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c8b07da-9af1-42fd-a6c8-4491d9a44d1c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c8b07da-9af1-42fd-a6c8-4491d9a44d1c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@720e0115 2021-05-04T01:07:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c8b07da-9af1-42fd-a6c8-4491d9a44d1c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:14.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c8b07da-9af1-42fd-a6c8-4491d9a44d1c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:17.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7ae0c10-de2b-485e-8690-e9523d3b3296 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:17.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7ae0c10-de2b-485e-8690-e9523d3b3296 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18f3b7c3 2021-05-04T01:07:17.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7ae0c10-de2b-485e-8690-e9523d3b3296 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:17.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7ae0c10-de2b-485e-8690-e9523d3b3296 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:20.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:07:20.189Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:07:20.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebb7f3fa-202b-434c-a83b-e9965ec54ecd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:20.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebb7f3fa-202b-434c-a83b-e9965ec54ecd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@722d653f 2021-05-04T01:07:20.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebb7f3fa-202b-434c-a83b-e9965ec54ecd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:20.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebb7f3fa-202b-434c-a83b-e9965ec54ecd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:23.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6347d11b-7306-421a-ae93-f5506b6f485a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:23.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6347d11b-7306-421a-ae93-f5506b6f485a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71253081 2021-05-04T01:07:23.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6347d11b-7306-421a-ae93-f5506b6f485a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:23.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6347d11b-7306-421a-ae93-f5506b6f485a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:26.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d010886-0c3e-40f0-98b0-f999cc5275f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:26.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d010886-0c3e-40f0-98b0-f999cc5275f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73a18119 2021-05-04T01:07:26.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d010886-0c3e-40f0-98b0-f999cc5275f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:26.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d010886-0c3e-40f0-98b0-f999cc5275f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09c6c47f-509d-4ce3-9f2f-b8941d9f8f3c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09c6c47f-509d-4ce3-9f2f-b8941d9f8f3c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76e96d01 2021-05-04T01:07:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09c6c47f-509d-4ce3-9f2f-b8941d9f8f3c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:29.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09c6c47f-509d-4ce3-9f2f-b8941d9f8f3c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7626c2da-ef23-43c3-a528-2428f077737b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7626c2da-ef23-43c3-a528-2428f077737b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ff4fa7e 2021-05-04T01:07:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7626c2da-ef23-43c3-a528-2428f077737b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:32.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7626c2da-ef23-43c3-a528-2428f077737b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:07:35.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:07:35.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=408ecd83-c791-4dbc-ad89-15f8ea2ea3e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:35.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=408ecd83-c791-4dbc-ad89-15f8ea2ea3e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@378d6e39 2021-05-04T01:07:35.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=408ecd83-c791-4dbc-ad89-15f8ea2ea3e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:35.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=408ecd83-c791-4dbc-ad89-15f8ea2ea3e5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:07:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:07:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:07:38.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b836d429-fb38-49ee-ad6f-908d74c5e5bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:38.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b836d429-fb38-49ee-ad6f-908d74c5e5bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@485b9898 2021-05-04T01:07:38.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b836d429-fb38-49ee-ad6f-908d74c5e5bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:38.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b836d429-fb38-49ee-ad6f-908d74c5e5bb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:41.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff73ecaa-9f5a-4ab8-b534-54ee8162f154 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:41.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff73ecaa-9f5a-4ab8-b534-54ee8162f154 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75e765f7 2021-05-04T01:07:41.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff73ecaa-9f5a-4ab8-b534-54ee8162f154 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:41.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff73ecaa-9f5a-4ab8-b534-54ee8162f154 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:44.424Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=248d1a08-167e-4e96-b275-322dd216a864 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:44.424Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=248d1a08-167e-4e96-b275-322dd216a864 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d08563f 2021-05-04T01:07:44.424Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=248d1a08-167e-4e96-b275-322dd216a864 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:44.424Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=248d1a08-167e-4e96-b275-322dd216a864 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:47.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d54b4c21-edf4-4785-9d6d-4b23bf24140c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:47.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d54b4c21-edf4-4785-9d6d-4b23bf24140c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61280b01 2021-05-04T01:07:47.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d54b4c21-edf4-4785-9d6d-4b23bf24140c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:47.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d54b4c21-edf4-4785-9d6d-4b23bf24140c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:07:50.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:07:50.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=759ed207-d570-42af-8f0b-dc1cb4eab792 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:50.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=759ed207-d570-42af-8f0b-dc1cb4eab792 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75a3d0e4 2021-05-04T01:07:50.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=759ed207-d570-42af-8f0b-dc1cb4eab792 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:50.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=759ed207-d570-42af-8f0b-dc1cb4eab792 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:52.041Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T01:07:53.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=297bebe6-fc96-4552-9901-5793f8161e14 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:53.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=297bebe6-fc96-4552-9901-5793f8161e14 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ba5db7a 2021-05-04T01:07:53.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=297bebe6-fc96-4552-9901-5793f8161e14 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:53.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=297bebe6-fc96-4552-9901-5793f8161e14 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:56.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=957404e6-3d73-46ab-ab35-ef634f1fbea7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:56.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=957404e6-3d73-46ab-ab35-ef634f1fbea7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4463cb9f 2021-05-04T01:07:56.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=957404e6-3d73-46ab-ab35-ef634f1fbea7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:56.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=957404e6-3d73-46ab-ab35-ef634f1fbea7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:07:59.450Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a4b752e-6340-4040-a8d5-8029e98dd54c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:07:59.450Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a4b752e-6340-4040-a8d5-8029e98dd54c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65b2b011 2021-05-04T01:07:59.450Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a4b752e-6340-4040-a8d5-8029e98dd54c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:07:59.450Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a4b752e-6340-4040-a8d5-8029e98dd54c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:02.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=114a9829-ef10-4281-a1d2-7cfdb44fe014 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:02.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=114a9829-ef10-4281-a1d2-7cfdb44fe014 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e7d78b3 2021-05-04T01:08:02.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=114a9829-ef10-4281-a1d2-7cfdb44fe014 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:02.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=114a9829-ef10-4281-a1d2-7cfdb44fe014 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:08:05.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:08:05.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cc639f0-ce67-4c38-b81b-2cddb1d56200 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:05.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cc639f0-ce67-4c38-b81b-2cddb1d56200 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b160c0d 2021-05-04T01:08:05.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cc639f0-ce67-4c38-b81b-2cddb1d56200 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:05.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cc639f0-ce67-4c38-b81b-2cddb1d56200 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:08.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4f2d872-49f3-4b39-998c-5fa057d8cb7b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:08.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4f2d872-49f3-4b39-998c-5fa057d8cb7b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@417303c3 2021-05-04T01:08:08.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4f2d872-49f3-4b39-998c-5fa057d8cb7b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:08.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4f2d872-49f3-4b39-998c-5fa057d8cb7b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:11.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bd399f5-9fd1-411d-a629-38af6aa60088 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:11.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bd399f5-9fd1-411d-a629-38af6aa60088 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@313101c4 2021-05-04T01:08:11.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bd399f5-9fd1-411d-a629-38af6aa60088 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:11.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bd399f5-9fd1-411d-a629-38af6aa60088 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:14.781Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc8bbd16-6f13-473d-9a51-9f98f928949b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:14.781Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc8bbd16-6f13-473d-9a51-9f98f928949b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@546c5fb1 2021-05-04T01:08:14.781Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc8bbd16-6f13-473d-9a51-9f98f928949b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:14.781Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc8bbd16-6f13-473d-9a51-9f98f928949b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=667f75d1-ff70-4202-916e-cfd16c0d6142 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=667f75d1-ff70-4202-916e-cfd16c0d6142 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fb15755 2021-05-04T01:08:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=667f75d1-ff70-4202-916e-cfd16c0d6142 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:17.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=667f75d1-ff70-4202-916e-cfd16c0d6142 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:08:20.174Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:08:20.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4c51d0-6490-4377-9124-aa3fb64ab6e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:20.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4c51d0-6490-4377-9124-aa3fb64ab6e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26052e7b 2021-05-04T01:08:20.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4c51d0-6490-4377-9124-aa3fb64ab6e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:20.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4c51d0-6490-4377-9124-aa3fb64ab6e7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f54da8d7-4140-4466-a644-e3f42c3fcaad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:23.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f54da8d7-4140-4466-a644-e3f42c3fcaad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f83245e 2021-05-04T01:08:23.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f54da8d7-4140-4466-a644-e3f42c3fcaad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:23.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f54da8d7-4140-4466-a644-e3f42c3fcaad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:26.347Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=354b9981-61d9-49c0-a194-e38184776c28 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:26.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=354b9981-61d9-49c0-a194-e38184776c28 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29a461c8 2021-05-04T01:08:26.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=354b9981-61d9-49c0-a194-e38184776c28 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:26.348Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=354b9981-61d9-49c0-a194-e38184776c28 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:29.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5603881-5d92-4898-94fe-b4c25f72beea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:29.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5603881-5d92-4898-94fe-b4c25f72beea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c5a8056 2021-05-04T01:08:29.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5603881-5d92-4898-94fe-b4c25f72beea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:29.446Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5603881-5d92-4898-94fe-b4c25f72beea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:32.286Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d007a80-d169-4bbe-b772-ae75bdb43bf5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:32.286Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d007a80-d169-4bbe-b772-ae75bdb43bf5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53a30b50 2021-05-04T01:08:32.286Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d007a80-d169-4bbe-b772-ae75bdb43bf5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:32.286Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d007a80-d169-4bbe-b772-ae75bdb43bf5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:35.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:08:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:08:35.292Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=3d007a80-d169-4bbe-b772-ae75bdb43bf5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:08:35.294Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=3d007a80-d169-4bbe-b772-ae75bdb43bf5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:08:35.294Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3d007a80-d169-4bbe-b772-ae75bdb43bf5 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:08:35.294Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3d007a80-d169-4bbe-b772-ae75bdb43bf5 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:08:36.913Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cfd55d6-dd75-4205-befb-2d567938ba90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:36.913Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cfd55d6-dd75-4205-befb-2d567938ba90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@550464ae 2021-05-04T01:08:36.913Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cfd55d6-dd75-4205-befb-2d567938ba90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:36.913Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cfd55d6-dd75-4205-befb-2d567938ba90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:08:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:08:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:08:38.454Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3cfd55d6-dd75-4205-befb-2d567938ba90 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:08:38.455Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3cfd55d6-dd75-4205-befb-2d567938ba90 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:08:39.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbddf355-bb47-478d-97c3-ca0070efcd25 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:39.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbddf355-bb47-478d-97c3-ca0070efcd25 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57f49665 2021-05-04T01:08:39.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbddf355-bb47-478d-97c3-ca0070efcd25 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:39.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbddf355-bb47-478d-97c3-ca0070efcd25 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=190324f5-23fb-4b5a-93ac-5051ddddbb18 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=190324f5-23fb-4b5a-93ac-5051ddddbb18 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1cc6fe40 2021-05-04T01:08:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=190324f5-23fb-4b5a-93ac-5051ddddbb18 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=190324f5-23fb-4b5a-93ac-5051ddddbb18 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:44.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b01eea1e-0006-42e9-81fe-9174a6406a82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:44.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b01eea1e-0006-42e9-81fe-9174a6406a82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a11c862 2021-05-04T01:08:44.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b01eea1e-0006-42e9-81fe-9174a6406a82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:44.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b01eea1e-0006-42e9-81fe-9174a6406a82 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:47.086Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba0ab7f7-e6cb-4ebb-833f-df7f9b7822ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:47.086Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba0ab7f7-e6cb-4ebb-833f-df7f9b7822ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12e2930a 2021-05-04T01:08:47.086Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba0ab7f7-e6cb-4ebb-833f-df7f9b7822ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:47.087Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba0ab7f7-e6cb-4ebb-833f-df7f9b7822ca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:08:50.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:08:50.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a0f75be-acf3-4b08-83be-c8531f2082f2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:50.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a0f75be-acf3-4b08-83be-c8531f2082f2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61d6788d 2021-05-04T01:08:50.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a0f75be-acf3-4b08-83be-c8531f2082f2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:50.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a0f75be-acf3-4b08-83be-c8531f2082f2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:52.050Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling \t--> HTTP/1.1 200 OK 2021-05-04T01:08:53.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5d0cc94-d55b-4a06-9f58-da1f7be834e3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:53.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5d0cc94-d55b-4a06-9f58-da1f7be834e3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ac9ae09 2021-05-04T01:08:53.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5d0cc94-d55b-4a06-9f58-da1f7be834e3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:53.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5d0cc94-d55b-4a06-9f58-da1f7be834e3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:56.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=813ed3aa-fda9-401d-8df8-bfd21eb3d579 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:56.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=813ed3aa-fda9-401d-8df8-bfd21eb3d579 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2eff3dea 2021-05-04T01:08:56.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=813ed3aa-fda9-401d-8df8-bfd21eb3d579 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:56.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=813ed3aa-fda9-401d-8df8-bfd21eb3d579 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:08:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f132c4-a10a-48a5-8568-e02238354e77 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:08:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f132c4-a10a-48a5-8568-e02238354e77 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@280391fc 2021-05-04T01:08:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f132c4-a10a-48a5-8568-e02238354e77 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:08:59.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f132c4-a10a-48a5-8568-e02238354e77 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:02.449Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e488caf-29cb-48d1-9755-e503183262c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:02.449Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e488caf-29cb-48d1-9755-e503183262c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a439661 2021-05-04T01:09:02.449Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e488caf-29cb-48d1-9755-e503183262c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:02.449Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e488caf-29cb-48d1-9755-e503183262c2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:09:05.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:09:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31b2e8f8-aa33-4d4b-a61b-d97bd680782f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31b2e8f8-aa33-4d4b-a61b-d97bd680782f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7d7dc6c6 2021-05-04T01:09:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31b2e8f8-aa33-4d4b-a61b-d97bd680782f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:05.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31b2e8f8-aa33-4d4b-a61b-d97bd680782f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:08.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c9c3ab3-2717-46a9-a50d-d1126b025ba3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:08.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c9c3ab3-2717-46a9-a50d-d1126b025ba3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a3aa4ee 2021-05-04T01:09:08.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c9c3ab3-2717-46a9-a50d-d1126b025ba3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:08.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c9c3ab3-2717-46a9-a50d-d1126b025ba3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:11.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fee8113c-3bd9-4a53-98dc-c034b16dbd93 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:11.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fee8113c-3bd9-4a53-98dc-c034b16dbd93 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e824c24 2021-05-04T01:09:11.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fee8113c-3bd9-4a53-98dc-c034b16dbd93 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:11.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fee8113c-3bd9-4a53-98dc-c034b16dbd93 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:14.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce1e48f8-c259-4fce-9cab-882791364a56 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:14.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce1e48f8-c259-4fce-9cab-882791364a56 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7814310b 2021-05-04T01:09:14.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce1e48f8-c259-4fce-9cab-882791364a56 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:14.520Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce1e48f8-c259-4fce-9cab-882791364a56 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:17.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1efaafcc-79a7-46da-8e64-9197cbe34457 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:17.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1efaafcc-79a7-46da-8e64-9197cbe34457 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b4c9339 2021-05-04T01:09:17.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1efaafcc-79a7-46da-8e64-9197cbe34457 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:17.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1efaafcc-79a7-46da-8e64-9197cbe34457 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:09:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=701b7114-f204-470a-860f-3230df8f166b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=701b7114-f204-470a-860f-3230df8f166b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@142e476a 2021-05-04T01:09:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=701b7114-f204-470a-860f-3230df8f166b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:20.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=701b7114-f204-470a-860f-3230df8f166b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:21.088Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-05-04T01:09:23.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d537ddb-24b3-4b9f-9610-bee5f1516925 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:23.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d537ddb-24b3-4b9f-9610-bee5f1516925 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1841daa7 2021-05-04T01:09:23.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d537ddb-24b3-4b9f-9610-bee5f1516925 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:23.421Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d537ddb-24b3-4b9f-9610-bee5f1516925 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:26.428Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed50e0b2-95c3-474c-8063-b8ad52249c2c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:26.428Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed50e0b2-95c3-474c-8063-b8ad52249c2c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41842ccd 2021-05-04T01:09:26.428Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed50e0b2-95c3-474c-8063-b8ad52249c2c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:26.428Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed50e0b2-95c3-474c-8063-b8ad52249c2c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f95a3f4-acb7-4046-a62d-e7846ef18952 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f95a3f4-acb7-4046-a62d-e7846ef18952 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4edeaa9f 2021-05-04T01:09:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f95a3f4-acb7-4046-a62d-e7846ef18952 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f95a3f4-acb7-4046-a62d-e7846ef18952 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:31.947Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13faedc9-2dc6-4a70-846f-37a560218139 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:31.947Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13faedc9-2dc6-4a70-846f-37a560218139 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@282db139 2021-05-04T01:09:31.948Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13faedc9-2dc6-4a70-846f-37a560218139 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:31.948Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13faedc9-2dc6-4a70-846f-37a560218139 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:09:35.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=540b70c6-542f-437d-9816-95f40f1d516d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:35.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=540b70c6-542f-437d-9816-95f40f1d516d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3eafc279 2021-05-04T01:09:35.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=540b70c6-542f-437d-9816-95f40f1d516d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:35.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=540b70c6-542f-437d-9816-95f40f1d516d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:09:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:09:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:09:38.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e19b9be-d403-4481-bc06-f27ea1c87059 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e19b9be-d403-4481-bc06-f27ea1c87059 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e2d6325 2021-05-04T01:09:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e19b9be-d403-4481-bc06-f27ea1c87059 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:38.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e19b9be-d403-4481-bc06-f27ea1c87059 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:41.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96eef435-e638-43a5-95c0-74d710a50973 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:41.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96eef435-e638-43a5-95c0-74d710a50973 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@659cb02d 2021-05-04T01:09:41.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96eef435-e638-43a5-95c0-74d710a50973 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:41.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96eef435-e638-43a5-95c0-74d710a50973 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:43.169Z [UEB-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServiceName=uebHealthCheck TargetServiceName=null ErrorCode=200 TargetEntity=uebHealthCheck ASDC Backend probably lost connectivity to UEB Cluster. Please check the logs for more information. 2021-05-04T01:09:44.394Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=def78754-276d-4096-813f-c94cab9eacb0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:44.394Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=def78754-276d-4096-813f-c94cab9eacb0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@207d61de 2021-05-04T01:09:44.394Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=def78754-276d-4096-813f-c94cab9eacb0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:44.394Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=def78754-276d-4096-813f-c94cab9eacb0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:45.212Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=def78754-276d-4096-813f-c94cab9eacb0 ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component DE state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:09:45.212Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=def78754-276d-4096-813f-c94cab9eacb0 ErrorCategory=INFO ServiceName=BE Component DE state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:09:47.039Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08022905-9741-4e88-8a8f-fe0eb8390939 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:47.039Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08022905-9741-4e88-8a8f-fe0eb8390939 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6081d3a2 2021-05-04T01:09:47.039Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08022905-9741-4e88-8a8f-fe0eb8390939 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:47.039Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08022905-9741-4e88-8a8f-fe0eb8390939 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:50.096Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac0eb1ef-82ab-4024-869f-79b55d02ae8f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:50.097Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac0eb1ef-82ab-4024-869f-79b55d02ae8f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5580a0dc 2021-05-04T01:09:50.097Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac0eb1ef-82ab-4024-869f-79b55d02ae8f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:50.097Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac0eb1ef-82ab-4024-869f-79b55d02ae8f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c3b75e7-80bd-41b7-b1d4-548e178f6400 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c3b75e7-80bd-41b7-b1d4-548e178f6400 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@551d6f99 2021-05-04T01:09:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c3b75e7-80bd-41b7-b1d4-548e178f6400 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:53.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c3b75e7-80bd-41b7-b1d4-548e178f6400 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd37dc94-cdbb-4c78-a01c-ca354bcbf11a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd37dc94-cdbb-4c78-a01c-ca354bcbf11a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ea59775 2021-05-04T01:09:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd37dc94-cdbb-4c78-a01c-ca354bcbf11a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:56.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd37dc94-cdbb-4c78-a01c-ca354bcbf11a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:09:59.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f20771c5-b81c-4cdc-9ad6-8bff985f79d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:09:59.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f20771c5-b81c-4cdc-9ad6-8bff985f79d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70d74bc0 2021-05-04T01:09:59.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f20771c5-b81c-4cdc-9ad6-8bff985f79d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:09:59.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f20771c5-b81c-4cdc-9ad6-8bff985f79d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:02.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2ebb1ea-05b4-4a35-8846-e2cde056f6bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:02.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2ebb1ea-05b4-4a35-8846-e2cde056f6bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20018346 2021-05-04T01:10:02.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2ebb1ea-05b4-4a35-8846-e2cde056f6bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:02.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2ebb1ea-05b4-4a35-8846-e2cde056f6bb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:05.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75059950-31d4-4433-8794-8c9f56a259f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:05.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75059950-31d4-4433-8794-8c9f56a259f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e5e4178 2021-05-04T01:10:05.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75059950-31d4-4433-8794-8c9f56a259f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:05.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75059950-31d4-4433-8794-8c9f56a259f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:08.075Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01eoz1n0nvs3phs7a7darns63t288 2021-05-04T01:10:08.075Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node012fw0ykx1eb3otmho64kfreys295 2021-05-04T01:10:08.075Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01rhylrl9tl9i81nd33q1e8s560290 2021-05-04T01:10:08.075Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node019jokubowqbl1we5fnypx1ghf289 2021-05-04T01:10:08.075Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01hagqc9raqfk0infcqho3ozll283 2021-05-04T01:10:08.076Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01n1nsd6jo94zqz3csdqrnxcv5296 2021-05-04T01:10:08.076Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0rvl6qr7umi74nqqncrh9swaz287 2021-05-04T01:10:08.076Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01snmxmljag4761wisqq4m6s888284 2021-05-04T01:10:08.076Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ab14uexr6ty71x5y5poel43j8286 2021-05-04T01:10:08.076Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0fvlv1gr38xogc5smmoduamwt285 2021-05-04T01:10:08.076Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0a1yfy78wtpgy1wex19iwnbuj5293 2021-05-04T01:10:08.076Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node034u1w0mqki9h1j0we22adxbe8292 2021-05-04T01:10:09.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c416202-a468-467e-aef8-69db8e9f4fc1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:09.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c416202-a468-467e-aef8-69db8e9f4fc1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c7263f9 2021-05-04T01:10:09.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c416202-a468-467e-aef8-69db8e9f4fc1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:09.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c416202-a468-467e-aef8-69db8e9f4fc1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:11.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0d7f744-ea77-43a2-975f-ba097c377f44 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:11.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0d7f744-ea77-43a2-975f-ba097c377f44 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6631a4fc 2021-05-04T01:10:11.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0d7f744-ea77-43a2-975f-ba097c377f44 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:11.198Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0d7f744-ea77-43a2-975f-ba097c377f44 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bb37efd-14ae-4813-b7f5-382a7e1b0baa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bb37efd-14ae-4813-b7f5-382a7e1b0baa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a0610cf 2021-05-04T01:10:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bb37efd-14ae-4813-b7f5-382a7e1b0baa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bb37efd-14ae-4813-b7f5-382a7e1b0baa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:17.137Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d410a6d-07da-42f7-952a-bd4fe18592c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:17.138Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d410a6d-07da-42f7-952a-bd4fe18592c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26bfdd76 2021-05-04T01:10:17.138Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d410a6d-07da-42f7-952a-bd4fe18592c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:17.138Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d410a6d-07da-42f7-952a-bd4fe18592c1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:20.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f416dc1-b0e6-44f4-9c95-51436747bccb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:20.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f416dc1-b0e6-44f4-9c95-51436747bccb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@196d1ff3 2021-05-04T01:10:20.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f416dc1-b0e6-44f4-9c95-51436747bccb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:20.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f416dc1-b0e6-44f4-9c95-51436747bccb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:23.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cde708f3-d0de-4ced-a7ad-f4f119ab1bc5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:23.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cde708f3-d0de-4ced-a7ad-f4f119ab1bc5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72105e66 2021-05-04T01:10:23.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cde708f3-d0de-4ced-a7ad-f4f119ab1bc5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:23.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cde708f3-d0de-4ced-a7ad-f4f119ab1bc5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:26.076Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d30c5e45-8676-493a-bced-d92a315d9b32 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:26.077Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d30c5e45-8676-493a-bced-d92a315d9b32 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@86aa883 2021-05-04T01:10:26.077Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d30c5e45-8676-493a-bced-d92a315d9b32 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:26.077Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d30c5e45-8676-493a-bced-d92a315d9b32 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:29.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46162f81-6129-416e-aa27-01b98c451a4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46162f81-6129-416e-aa27-01b98c451a4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50e69095 2021-05-04T01:10:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46162f81-6129-416e-aa27-01b98c451a4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:29.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46162f81-6129-416e-aa27-01b98c451a4e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:32.173Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99f66130-8219-4573-82e5-415cd633149f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:32.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99f66130-8219-4573-82e5-415cd633149f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21859ffd 2021-05-04T01:10:32.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99f66130-8219-4573-82e5-415cd633149f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:32.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99f66130-8219-4573-82e5-415cd633149f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27ab27a2-03ee-4e3c-a09f-c36d53858335 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:35.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27ab27a2-03ee-4e3c-a09f-c36d53858335 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@511616c5 2021-05-04T01:10:35.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27ab27a2-03ee-4e3c-a09f-c36d53858335 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:35.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27ab27a2-03ee-4e3c-a09f-c36d53858335 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:38.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=134e97e0-c1aa-4902-8158-7cc77ae129f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:38.412Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=134e97e0-c1aa-4902-8158-7cc77ae129f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c6582d4 2021-05-04T01:10:38.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=134e97e0-c1aa-4902-8158-7cc77ae129f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:38.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=134e97e0-c1aa-4902-8158-7cc77ae129f1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:42.489Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a323b54-9735-4559-b4c8-40bab4c0eb56 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:42.489Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a323b54-9735-4559-b4c8-40bab4c0eb56 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68e3d9f5 2021-05-04T01:10:42.489Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a323b54-9735-4559-b4c8-40bab4c0eb56 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:42.489Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a323b54-9735-4559-b4c8-40bab4c0eb56 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:44.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7febde0d-ea29-47aa-a4ae-8388bdf9d54b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:44.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7febde0d-ea29-47aa-a4ae-8388bdf9d54b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6594c963 2021-05-04T01:10:44.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7febde0d-ea29-47aa-a4ae-8388bdf9d54b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:44.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7febde0d-ea29-47aa-a4ae-8388bdf9d54b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:47.025Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d56bd3e9-02b7-45a9-b5ee-2b28f68b4a71 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:47.025Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d56bd3e9-02b7-45a9-b5ee-2b28f68b4a71 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d54b2a 2021-05-04T01:10:47.025Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d56bd3e9-02b7-45a9-b5ee-2b28f68b4a71 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:47.025Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d56bd3e9-02b7-45a9-b5ee-2b28f68b4a71 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c911b3b-4fe3-450b-831e-16ac436b3481 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c911b3b-4fe3-450b-831e-16ac436b3481 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24861a46 2021-05-04T01:10:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c911b3b-4fe3-450b-831e-16ac436b3481 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:50.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c911b3b-4fe3-450b-831e-16ac436b3481 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:53.337Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a893050-58cc-4b17-95db-6aa6bbcb77c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:53.337Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a893050-58cc-4b17-95db-6aa6bbcb77c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@525c4352 2021-05-04T01:10:53.337Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a893050-58cc-4b17-95db-6aa6bbcb77c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:53.337Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a893050-58cc-4b17-95db-6aa6bbcb77c6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:56.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e97d1da5-d357-40a4-9081-eea031f72c89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:56.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e97d1da5-d357-40a4-9081-eea031f72c89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@11c057ab 2021-05-04T01:10:56.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e97d1da5-d357-40a4-9081-eea031f72c89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:56.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e97d1da5-d357-40a4-9081-eea031f72c89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:10:59.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c413a54-b5d9-4e84-ab60-99dba5a5d013 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:10:59.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c413a54-b5d9-4e84-ab60-99dba5a5d013 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3fe2d2a2 2021-05-04T01:10:59.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c413a54-b5d9-4e84-ab60-99dba5a5d013 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:10:59.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c413a54-b5d9-4e84-ab60-99dba5a5d013 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:02.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=666274cd-4865-483b-959b-c88b87c6f144 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:02.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=666274cd-4865-483b-959b-c88b87c6f144 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ba2ab32 2021-05-04T01:11:02.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=666274cd-4865-483b-959b-c88b87c6f144 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:02.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=666274cd-4865-483b-959b-c88b87c6f144 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:06.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d677fcd-9082-47d0-81f6-dd642760c47c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:06.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d677fcd-9082-47d0-81f6-dd642760c47c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bd43bc7 2021-05-04T01:11:06.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d677fcd-9082-47d0-81f6-dd642760c47c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:06.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d677fcd-9082-47d0-81f6-dd642760c47c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:08.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd041ce4-c1b6-4856-9438-98d4d6776dc7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:08.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd041ce4-c1b6-4856-9438-98d4d6776dc7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@83a4a19 2021-05-04T01:11:08.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd041ce4-c1b6-4856-9438-98d4d6776dc7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:08.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd041ce4-c1b6-4856-9438-98d4d6776dc7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:11.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf464a62-cdd5-4fc7-9750-1c079a4c709e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:11.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf464a62-cdd5-4fc7-9750-1c079a4c709e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2164ee7e 2021-05-04T01:11:11.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf464a62-cdd5-4fc7-9750-1c079a4c709e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:11.212Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf464a62-cdd5-4fc7-9750-1c079a4c709e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:14.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbe3c2a-0765-40bc-9a30-62f0957b7888 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:14.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbe3c2a-0765-40bc-9a30-62f0957b7888 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1fa13a5b 2021-05-04T01:11:14.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbe3c2a-0765-40bc-9a30-62f0957b7888 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:14.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbe3c2a-0765-40bc-9a30-62f0957b7888 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:17.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03c05842-e0c8-44d3-bbe1-8cfafabaa9a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:17.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03c05842-e0c8-44d3-bbe1-8cfafabaa9a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2511206b 2021-05-04T01:11:17.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03c05842-e0c8-44d3-bbe1-8cfafabaa9a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:17.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03c05842-e0c8-44d3-bbe1-8cfafabaa9a2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:20.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e52cc18-ea5e-4948-948f-81bbff4f904e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:20.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e52cc18-ea5e-4948-948f-81bbff4f904e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f21cc52 2021-05-04T01:11:20.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e52cc18-ea5e-4948-948f-81bbff4f904e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:20.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e52cc18-ea5e-4948-948f-81bbff4f904e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:23.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ed21c48-ed93-4331-a56a-51d58835590f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:23.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ed21c48-ed93-4331-a56a-51d58835590f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79b0cad6 2021-05-04T01:11:23.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ed21c48-ed93-4331-a56a-51d58835590f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:23.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ed21c48-ed93-4331-a56a-51d58835590f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:26.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c380f06b-f162-4339-85d0-53403e7a7319 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:26.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c380f06b-f162-4339-85d0-53403e7a7319 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f554459 2021-05-04T01:11:26.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c380f06b-f162-4339-85d0-53403e7a7319 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:26.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c380f06b-f162-4339-85d0-53403e7a7319 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:29.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35a3dfd4-bbc2-4556-a50c-39257034a3b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:29.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35a3dfd4-bbc2-4556-a50c-39257034a3b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43eb23a0 2021-05-04T01:11:29.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35a3dfd4-bbc2-4556-a50c-39257034a3b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:29.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35a3dfd4-bbc2-4556-a50c-39257034a3b9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:32.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf53f90-28ad-4522-8635-80840739ce9e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:32.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf53f90-28ad-4522-8635-80840739ce9e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32d924d3 2021-05-04T01:11:32.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf53f90-28ad-4522-8635-80840739ce9e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:32.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf53f90-28ad-4522-8635-80840739ce9e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:35.405Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654c629c-567b-46b9-bc29-cad8a4c36446 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:35.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654c629c-567b-46b9-bc29-cad8a4c36446 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d40ae24 2021-05-04T01:11:35.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654c629c-567b-46b9-bc29-cad8a4c36446 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:35.406Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=654c629c-567b-46b9-bc29-cad8a4c36446 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:38.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de8b84a9-1214-4409-8dab-c212a2bf1956 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:38.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de8b84a9-1214-4409-8dab-c212a2bf1956 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8f24c59 2021-05-04T01:11:38.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de8b84a9-1214-4409-8dab-c212a2bf1956 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:38.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de8b84a9-1214-4409-8dab-c212a2bf1956 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:41.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fef4392-33e5-4509-a3fd-1a56d6a4aa97 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:41.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fef4392-33e5-4509-a3fd-1a56d6a4aa97 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@129e98c4 2021-05-04T01:11:41.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fef4392-33e5-4509-a3fd-1a56d6a4aa97 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:41.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fef4392-33e5-4509-a3fd-1a56d6a4aa97 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:44.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56f5523b-2906-4a88-a37d-850a55c41cfc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:44.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56f5523b-2906-4a88-a37d-850a55c41cfc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@691ac9f8 2021-05-04T01:11:44.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56f5523b-2906-4a88-a37d-850a55c41cfc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:44.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56f5523b-2906-4a88-a37d-850a55c41cfc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:47.183Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39734a5-df76-4a78-b975-ec89ac688acc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:47.183Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39734a5-df76-4a78-b975-ec89ac688acc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21e36b71 2021-05-04T01:11:47.183Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39734a5-df76-4a78-b975-ec89ac688acc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:47.183Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39734a5-df76-4a78-b975-ec89ac688acc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:50.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa605da4-a137-4362-bc80-bb9ada0938a4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:50.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa605da4-a137-4362-bc80-bb9ada0938a4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3be97b90 2021-05-04T01:11:50.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa605da4-a137-4362-bc80-bb9ada0938a4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:50.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa605da4-a137-4362-bc80-bb9ada0938a4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:53.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=338b31c2-a7bd-4640-9a62-47d1857eec99 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:11:53.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=338b31c2-a7bd-4640-9a62-47d1857eec99 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5dd78c91 2021-05-04T01:11:53.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=338b31c2-a7bd-4640-9a62-47d1857eec99 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:11:53.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=338b31c2-a7bd-4640-9a62-47d1857eec99 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:11:56.463Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=338b31c2-a7bd-4640-9a62-47d1857eec99 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:11:56.464Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=338b31c2-a7bd-4640-9a62-47d1857eec99 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:11:56.464Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=338b31c2-a7bd-4640-9a62-47d1857eec99 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:11:56.464Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=338b31c2-a7bd-4640-9a62-47d1857eec99 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:11:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:11:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:12:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:12:02.450Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:12:02.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc05230c-854c-45eb-9596-e922d9342b82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:02.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc05230c-854c-45eb-9596-e922d9342b82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62bdcbda 2021-05-04T01:12:02.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc05230c-854c-45eb-9596-e922d9342b82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:02.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc05230c-854c-45eb-9596-e922d9342b82 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:03.055Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=dc05230c-854c-45eb-9596-e922d9342b82 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:12:03.055Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=dc05230c-854c-45eb-9596-e922d9342b82 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:12:03.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4db2a9a1-0c5e-4885-85da-254f87f4bc03 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:03.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4db2a9a1-0c5e-4885-85da-254f87f4bc03 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28fa5973 2021-05-04T01:12:03.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4db2a9a1-0c5e-4885-85da-254f87f4bc03 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:03.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4db2a9a1-0c5e-4885-85da-254f87f4bc03 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:05.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aae79899-5d89-4539-9d40-9ca547e86903 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:05.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aae79899-5d89-4539-9d40-9ca547e86903 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4928ebf9 2021-05-04T01:12:05.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aae79899-5d89-4539-9d40-9ca547e86903 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:05.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aae79899-5d89-4539-9d40-9ca547e86903 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:06.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bc8eb1e-2102-4fcd-b061-97dc8eab18fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:06.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bc8eb1e-2102-4fcd-b061-97dc8eab18fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45c71a1d 2021-05-04T01:12:06.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bc8eb1e-2102-4fcd-b061-97dc8eab18fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:06.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bc8eb1e-2102-4fcd-b061-97dc8eab18fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20a0bea9-a176-4d9d-9800-a23b73383c25 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20a0bea9-a176-4d9d-9800-a23b73383c25 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61bc39ce 2021-05-04T01:12:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20a0bea9-a176-4d9d-9800-a23b73383c25 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20a0bea9-a176-4d9d-9800-a23b73383c25 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:11.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b4eec6c-ed8a-4111-95e4-af6bbb09d39f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:11.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b4eec6c-ed8a-4111-95e4-af6bbb09d39f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b77c54b 2021-05-04T01:12:11.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b4eec6c-ed8a-4111-95e4-af6bbb09d39f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:11.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b4eec6c-ed8a-4111-95e4-af6bbb09d39f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:14.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20be5f3f-82ca-49a0-b145-2db9e9171aa6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:14.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20be5f3f-82ca-49a0-b145-2db9e9171aa6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@150e82ff 2021-05-04T01:12:14.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20be5f3f-82ca-49a0-b145-2db9e9171aa6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:14.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20be5f3f-82ca-49a0-b145-2db9e9171aa6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:17.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e41ce55-2dcb-4543-9728-acaa218a52e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:17.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e41ce55-2dcb-4543-9728-acaa218a52e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@686475b1 2021-05-04T01:12:17.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e41ce55-2dcb-4543-9728-acaa218a52e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:17.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e41ce55-2dcb-4543-9728-acaa218a52e7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:20.550Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34d428c1-02ab-4c65-8676-5d1f649badb6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:20.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34d428c1-02ab-4c65-8676-5d1f649badb6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6896533a 2021-05-04T01:12:20.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34d428c1-02ab-4c65-8676-5d1f649badb6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:20.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34d428c1-02ab-4c65-8676-5d1f649badb6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:23.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb51422f-27ff-45a9-9711-cadf45ce2e6d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:23.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb51422f-27ff-45a9-9711-cadf45ce2e6d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4dee9efe 2021-05-04T01:12:23.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb51422f-27ff-45a9-9711-cadf45ce2e6d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:23.606Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb51422f-27ff-45a9-9711-cadf45ce2e6d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:26.452Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7266e75c-29c8-42f5-8500-59eb24bb469e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:26.452Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7266e75c-29c8-42f5-8500-59eb24bb469e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c16f4c6 2021-05-04T01:12:26.452Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7266e75c-29c8-42f5-8500-59eb24bb469e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:26.452Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7266e75c-29c8-42f5-8500-59eb24bb469e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:29.345Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e398a2c-3e3f-4409-ab09-3bac1ecb02e8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:29.345Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e398a2c-3e3f-4409-ab09-3bac1ecb02e8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6714d508 2021-05-04T01:12:29.345Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e398a2c-3e3f-4409-ab09-3bac1ecb02e8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:29.345Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e398a2c-3e3f-4409-ab09-3bac1ecb02e8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:32.852Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d598c64-d138-45e4-9652-2571ec9d0d35 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:32.852Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d598c64-d138-45e4-9652-2571ec9d0d35 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3dc15553 2021-05-04T01:12:32.852Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d598c64-d138-45e4-9652-2571ec9d0d35 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:32.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d598c64-d138-45e4-9652-2571ec9d0d35 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c9472ad-e8fd-4130-b888-5c897e7a16b5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c9472ad-e8fd-4130-b888-5c897e7a16b5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e376b91 2021-05-04T01:12:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c9472ad-e8fd-4130-b888-5c897e7a16b5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c9472ad-e8fd-4130-b888-5c897e7a16b5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:38.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79423879-4174-452e-a387-a4a0f96c71cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:38.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79423879-4174-452e-a387-a4a0f96c71cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69ebc6c9 2021-05-04T01:12:38.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79423879-4174-452e-a387-a4a0f96c71cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:38.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79423879-4174-452e-a387-a4a0f96c71cb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:41.744Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87b81697-577b-4597-a722-835ff60e31d4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:41.745Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87b81697-577b-4597-a722-835ff60e31d4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@642c7c62 2021-05-04T01:12:41.745Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87b81697-577b-4597-a722-835ff60e31d4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:41.745Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87b81697-577b-4597-a722-835ff60e31d4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:44.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1396f86e-6495-4ee4-8c49-695247f3f61f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:44.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1396f86e-6495-4ee4-8c49-695247f3f61f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2924cc7b 2021-05-04T01:12:44.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1396f86e-6495-4ee4-8c49-695247f3f61f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:44.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1396f86e-6495-4ee4-8c49-695247f3f61f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3850c5b2-bf3e-4db9-94cb-2ccaffeb905a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3850c5b2-bf3e-4db9-94cb-2ccaffeb905a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4caa512d 2021-05-04T01:12:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3850c5b2-bf3e-4db9-94cb-2ccaffeb905a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3850c5b2-bf3e-4db9-94cb-2ccaffeb905a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:50.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=354ec119-4a52-4ba1-aa95-c43a65897a1f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:50.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=354ec119-4a52-4ba1-aa95-c43a65897a1f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7bc00bff 2021-05-04T01:12:50.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=354ec119-4a52-4ba1-aa95-c43a65897a1f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:50.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=354ec119-4a52-4ba1-aa95-c43a65897a1f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:53.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26341f4-100c-4f74-b7b2-c8a89f2fffd2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:53.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26341f4-100c-4f74-b7b2-c8a89f2fffd2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c08b789 2021-05-04T01:12:53.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26341f4-100c-4f74-b7b2-c8a89f2fffd2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:53.518Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26341f4-100c-4f74-b7b2-c8a89f2fffd2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71243455-3108-434b-bed6-c75eb627551d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71243455-3108-434b-bed6-c75eb627551d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73c0fb67 2021-05-04T01:12:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71243455-3108-434b-bed6-c75eb627551d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:56.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71243455-3108-434b-bed6-c75eb627551d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:12:59.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9c6e25f-b119-4173-b648-72ee7048805d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:12:59.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9c6e25f-b119-4173-b648-72ee7048805d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6dfdef0b 2021-05-04T01:12:59.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9c6e25f-b119-4173-b648-72ee7048805d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:12:59.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9c6e25f-b119-4173-b648-72ee7048805d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:05.384Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc53ec2a-777f-4f6f-a947-51ca343109d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:05.384Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc53ec2a-777f-4f6f-a947-51ca343109d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5bfb0a4b 2021-05-04T01:13:05.384Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc53ec2a-777f-4f6f-a947-51ca343109d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:05.385Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc53ec2a-777f-4f6f-a947-51ca343109d6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:08.389Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bc53ec2a-777f-4f6f-a947-51ca343109d6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:13:08.391Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bc53ec2a-777f-4f6f-a947-51ca343109d6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:13:08.391Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bc53ec2a-777f-4f6f-a947-51ca343109d6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:13:08.392Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bc53ec2a-777f-4f6f-a947-51ca343109d6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:13:10.493Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=578ddfe3-0d44-4a56-a292-0576c3b80db2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:10.493Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=578ddfe3-0d44-4a56-a292-0576c3b80db2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59fc3388 2021-05-04T01:13:10.493Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=578ddfe3-0d44-4a56-a292-0576c3b80db2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:10.493Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=578ddfe3-0d44-4a56-a292-0576c3b80db2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:11.172Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=578ddfe3-0d44-4a56-a292-0576c3b80db2 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:13:11.172Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=578ddfe3-0d44-4a56-a292-0576c3b80db2 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:13:12.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e3eabe2-4c6d-4a97-85c8-a69281244d84 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:12.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e3eabe2-4c6d-4a97-85c8-a69281244d84 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@aef890a 2021-05-04T01:13:12.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e3eabe2-4c6d-4a97-85c8-a69281244d84 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:12.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e3eabe2-4c6d-4a97-85c8-a69281244d84 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:13.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0199610-1991-4ef9-b491-71fc9a0ba42d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:13.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0199610-1991-4ef9-b491-71fc9a0ba42d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9c5734e 2021-05-04T01:13:13.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0199610-1991-4ef9-b491-71fc9a0ba42d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:13.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0199610-1991-4ef9-b491-71fc9a0ba42d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:14.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=664ea48b-a1f9-4d52-a855-57284c0ad9d1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:14.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=664ea48b-a1f9-4d52-a855-57284c0ad9d1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@319d718d 2021-05-04T01:13:14.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=664ea48b-a1f9-4d52-a855-57284c0ad9d1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:14.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=664ea48b-a1f9-4d52-a855-57284c0ad9d1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:18.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a63b3be7-851a-4781-935d-97501feb8bba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:18.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a63b3be7-851a-4781-935d-97501feb8bba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6652fd7d 2021-05-04T01:13:18.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a63b3be7-851a-4781-935d-97501feb8bba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:18.246Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a63b3be7-851a-4781-935d-97501feb8bba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:21.253Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a63b3be7-851a-4781-935d-97501feb8bba ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:13:21.253Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a63b3be7-851a-4781-935d-97501feb8bba ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:13:21.254Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a63b3be7-851a-4781-935d-97501feb8bba ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:13:21.254Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a63b3be7-851a-4781-935d-97501feb8bba ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:13:22.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6d67847-ce50-452e-916c-7d08dd50b879 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:22.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6d67847-ce50-452e-916c-7d08dd50b879 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10838f64 2021-05-04T01:13:22.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6d67847-ce50-452e-916c-7d08dd50b879 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:22.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6d67847-ce50-452e-916c-7d08dd50b879 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:23.358Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f6d67847-ce50-452e-916c-7d08dd50b879 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:13:23.359Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f6d67847-ce50-452e-916c-7d08dd50b879 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:13:24.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4c94019-d07b-4296-a1fb-09f960a8a172 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:24.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4c94019-d07b-4296-a1fb-09f960a8a172 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6835cfe2 2021-05-04T01:13:24.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4c94019-d07b-4296-a1fb-09f960a8a172 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:24.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4c94019-d07b-4296-a1fb-09f960a8a172 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:26.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35e49280-de39-44eb-b5fa-87300038c095 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:26.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35e49280-de39-44eb-b5fa-87300038c095 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@197d7e8c 2021-05-04T01:13:26.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35e49280-de39-44eb-b5fa-87300038c095 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:26.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35e49280-de39-44eb-b5fa-87300038c095 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:29.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2aa78cae-f4f8-4ca7-8c2b-95ecda291654 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:29.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2aa78cae-f4f8-4ca7-8c2b-95ecda291654 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15892a68 2021-05-04T01:13:29.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2aa78cae-f4f8-4ca7-8c2b-95ecda291654 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:29.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2aa78cae-f4f8-4ca7-8c2b-95ecda291654 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:32.177Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=888414a9-9009-403f-9424-e105cd7893a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:32.177Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=888414a9-9009-403f-9424-e105cd7893a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@513ac6f9 2021-05-04T01:13:32.177Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=888414a9-9009-403f-9424-e105cd7893a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:32.177Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=888414a9-9009-403f-9424-e105cd7893a0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9768c33-8a1b-47be-a6bd-ae870c7a4260 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9768c33-8a1b-47be-a6bd-ae870c7a4260 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@173659e3 2021-05-04T01:13:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9768c33-8a1b-47be-a6bd-ae870c7a4260 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9768c33-8a1b-47be-a6bd-ae870c7a4260 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:38.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d20e0a76-a6fd-453b-b8df-8cb86d7dd0ad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:38.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d20e0a76-a6fd-453b-b8df-8cb86d7dd0ad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@654dcc97 2021-05-04T01:13:38.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d20e0a76-a6fd-453b-b8df-8cb86d7dd0ad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:38.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d20e0a76-a6fd-453b-b8df-8cb86d7dd0ad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:41.650Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b7047e4-4478-449e-8b48-a240436f5eb5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:41.650Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b7047e4-4478-449e-8b48-a240436f5eb5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39cb603c 2021-05-04T01:13:41.650Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b7047e4-4478-449e-8b48-a240436f5eb5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:41.650Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b7047e4-4478-449e-8b48-a240436f5eb5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:45.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26d637f-a6b1-4e65-8b6b-4f72a1e4f0e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:45.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26d637f-a6b1-4e65-8b6b-4f72a1e4f0e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e4174f2 2021-05-04T01:13:45.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26d637f-a6b1-4e65-8b6b-4f72a1e4f0e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:45.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26d637f-a6b1-4e65-8b6b-4f72a1e4f0e1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:51.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f117f6a9-b4dc-4326-bb9a-98baa75948f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:51.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f117f6a9-b4dc-4326-bb9a-98baa75948f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b1dcb98 2021-05-04T01:13:51.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f117f6a9-b4dc-4326-bb9a-98baa75948f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:51.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f117f6a9-b4dc-4326-bb9a-98baa75948f1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:52.782Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c96f56cc-b909-421e-ab16-27f5d80cdf31 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:52.782Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c96f56cc-b909-421e-ab16-27f5d80cdf31 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10bf8518 2021-05-04T01:13:52.782Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c96f56cc-b909-421e-ab16-27f5d80cdf31 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:52.782Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c96f56cc-b909-421e-ab16-27f5d80cdf31 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:53.811Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0be2ce19-b4d3-4fca-9925-b97e99ac04fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:53.811Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0be2ce19-b4d3-4fca-9925-b97e99ac04fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7297cdb3 2021-05-04T01:13:53.811Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0be2ce19-b4d3-4fca-9925-b97e99ac04fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:53.811Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0be2ce19-b4d3-4fca-9925-b97e99ac04fe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:56.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92fde045-5f49-4cb6-b857-b8dcab65c8ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:56.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92fde045-5f49-4cb6-b857-b8dcab65c8ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4744d1f7 2021-05-04T01:13:56.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92fde045-5f49-4cb6-b857-b8dcab65c8ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:56.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92fde045-5f49-4cb6-b857-b8dcab65c8ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:13:59.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a0148bd-4a40-4347-bac3-0f1b3d31f5ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:13:59.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a0148bd-4a40-4347-bac3-0f1b3d31f5ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3270838c 2021-05-04T01:13:59.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a0148bd-4a40-4347-bac3-0f1b3d31f5ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:13:59.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a0148bd-4a40-4347-bac3-0f1b3d31f5ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:06.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b93179a-a1c2-4f0a-bcc4-97cd2a57f599 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:06.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b93179a-a1c2-4f0a-bcc4-97cd2a57f599 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e0b479f 2021-05-04T01:14:06.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b93179a-a1c2-4f0a-bcc4-97cd2a57f599 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:06.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b93179a-a1c2-4f0a-bcc4-97cd2a57f599 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:07.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=021c2400-bb3d-4314-8fa6-90f0ecc6eeda PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:07.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=021c2400-bb3d-4314-8fa6-90f0ecc6eeda PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12bde3b8 2021-05-04T01:14:07.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=021c2400-bb3d-4314-8fa6-90f0ecc6eeda PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:07.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=021c2400-bb3d-4314-8fa6-90f0ecc6eeda PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:08.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76223f9c-c3cf-4932-8d1a-7b2ef302fd2a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:08.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76223f9c-c3cf-4932-8d1a-7b2ef302fd2a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73974371 2021-05-04T01:14:08.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76223f9c-c3cf-4932-8d1a-7b2ef302fd2a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:08.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76223f9c-c3cf-4932-8d1a-7b2ef302fd2a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:11.397Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a604ec9-e801-412a-8e20-90cb5e9948d0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:11.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a604ec9-e801-412a-8e20-90cb5e9948d0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45cdd26c 2021-05-04T01:14:11.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a604ec9-e801-412a-8e20-90cb5e9948d0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:11.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a604ec9-e801-412a-8e20-90cb5e9948d0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=284ee847-553e-4ffd-920b-0d406c261b7e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=284ee847-553e-4ffd-920b-0d406c261b7e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41fe42f5 2021-05-04T01:14:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=284ee847-553e-4ffd-920b-0d406c261b7e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=284ee847-553e-4ffd-920b-0d406c261b7e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:17.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4495dd2-6c00-4515-a71f-e8e1dfdd0e3f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:17.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4495dd2-6c00-4515-a71f-e8e1dfdd0e3f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ca3d951 2021-05-04T01:14:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4495dd2-6c00-4515-a71f-e8e1dfdd0e3f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:17.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4495dd2-6c00-4515-a71f-e8e1dfdd0e3f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:20.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef0472d9-c31c-4818-9e65-c707c1ef426a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:20.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef0472d9-c31c-4818-9e65-c707c1ef426a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7566997f 2021-05-04T01:14:20.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef0472d9-c31c-4818-9e65-c707c1ef426a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:20.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef0472d9-c31c-4818-9e65-c707c1ef426a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:24.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ea7e548-c490-4f4a-9e05-ec422c976844 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:24.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ea7e548-c490-4f4a-9e05-ec422c976844 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e937b0d 2021-05-04T01:14:24.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ea7e548-c490-4f4a-9e05-ec422c976844 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:24.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ea7e548-c490-4f4a-9e05-ec422c976844 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:26.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90244cd0-eb8f-413f-ad1b-5a4035c0529a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:26.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90244cd0-eb8f-413f-ad1b-5a4035c0529a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d4963c0 2021-05-04T01:14:26.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90244cd0-eb8f-413f-ad1b-5a4035c0529a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:26.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90244cd0-eb8f-413f-ad1b-5a4035c0529a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:29.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd0f5a90-295b-483b-8364-08e379a736e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:29.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd0f5a90-295b-483b-8364-08e379a736e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f024669 2021-05-04T01:14:29.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd0f5a90-295b-483b-8364-08e379a736e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:29.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd0f5a90-295b-483b-8364-08e379a736e7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:33.738Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ca628bf-4e0b-4c67-9939-1ab96ef3b426 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:33.738Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ca628bf-4e0b-4c67-9939-1ab96ef3b426 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@eee9397 2021-05-04T01:14:33.738Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ca628bf-4e0b-4c67-9939-1ab96ef3b426 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:33.738Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ca628bf-4e0b-4c67-9939-1ab96ef3b426 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:36.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64e33622-69c1-4b5a-82db-f1b63d699c82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:36.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64e33622-69c1-4b5a-82db-f1b63d699c82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4be0da54 2021-05-04T01:14:36.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64e33622-69c1-4b5a-82db-f1b63d699c82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:36.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64e33622-69c1-4b5a-82db-f1b63d699c82 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:38.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2686f459-7613-41e4-b160-38a8cd2d6be4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:38.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2686f459-7613-41e4-b160-38a8cd2d6be4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d11f78b 2021-05-04T01:14:38.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2686f459-7613-41e4-b160-38a8cd2d6be4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:38.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2686f459-7613-41e4-b160-38a8cd2d6be4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:41.304Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2686f459-7613-41e4-b160-38a8cd2d6be4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:14:41.305Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2686f459-7613-41e4-b160-38a8cd2d6be4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:14:41.305Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2686f459-7613-41e4-b160-38a8cd2d6be4 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:14:41.305Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2686f459-7613-41e4-b160-38a8cd2d6be4 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:14:47.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db4c6313-4705-4f31-9dc1-fedaac8e5b23 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:47.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db4c6313-4705-4f31-9dc1-fedaac8e5b23 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30fdc54e 2021-05-04T01:14:47.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db4c6313-4705-4f31-9dc1-fedaac8e5b23 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:47.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db4c6313-4705-4f31-9dc1-fedaac8e5b23 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:49.363Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=db4c6313-4705-4f31-9dc1-fedaac8e5b23 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:14:49.363Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=db4c6313-4705-4f31-9dc1-fedaac8e5b23 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:14:50.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=479a90ea-acc5-48d0-b725-759d5bc4ad5e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:50.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=479a90ea-acc5-48d0-b725-759d5bc4ad5e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78acb8d 2021-05-04T01:14:50.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=479a90ea-acc5-48d0-b725-759d5bc4ad5e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:50.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=479a90ea-acc5-48d0-b725-759d5bc4ad5e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:51.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5bf021e-cfff-437d-b371-1be7f5529089 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:51.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5bf021e-cfff-437d-b371-1be7f5529089 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65c5d958 2021-05-04T01:14:51.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5bf021e-cfff-437d-b371-1be7f5529089 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:51.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5bf021e-cfff-437d-b371-1be7f5529089 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:52.809Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08da2918-1adf-4667-bbeb-b5e3760c3abc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:52.809Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08da2918-1adf-4667-bbeb-b5e3760c3abc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b4cb71b 2021-05-04T01:14:52.809Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08da2918-1adf-4667-bbeb-b5e3760c3abc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:52.809Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08da2918-1adf-4667-bbeb-b5e3760c3abc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:56.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1c97c2a-b460-40ce-b946-3d45aaaf94c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:56.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1c97c2a-b460-40ce-b946-3d45aaaf94c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d88707d 2021-05-04T01:14:56.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1c97c2a-b460-40ce-b946-3d45aaaf94c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:56.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1c97c2a-b460-40ce-b946-3d45aaaf94c6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:14:57.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f9ccae9-8aa3-4e3b-a362-10ec295161f0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:14:57.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f9ccae9-8aa3-4e3b-a362-10ec295161f0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c8f1ee2 2021-05-04T01:14:57.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f9ccae9-8aa3-4e3b-a362-10ec295161f0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:14:57.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f9ccae9-8aa3-4e3b-a362-10ec295161f0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:15:01.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2924f6db-5954-4716-aafa-000514afac94 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:01.345Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2924f6db-5954-4716-aafa-000514afac94 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47cb0932 2021-05-04T01:15:01.345Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2924f6db-5954-4716-aafa-000514afac94 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:01.345Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2924f6db-5954-4716-aafa-000514afac94 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:15:03.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dbadca1-dece-4204-83e4-66e379e76c68 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:03.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dbadca1-dece-4204-83e4-66e379e76c68 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19619abf 2021-05-04T01:15:03.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dbadca1-dece-4204-83e4-66e379e76c68 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:03.604Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dbadca1-dece-4204-83e4-66e379e76c68 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:15:06.845Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52f727de-b776-4c5a-bfe6-cfef75c00a45 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:06.845Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52f727de-b776-4c5a-bfe6-cfef75c00a45 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f40a76 2021-05-04T01:15:06.846Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52f727de-b776-4c5a-bfe6-cfef75c00a45 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:06.846Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52f727de-b776-4c5a-bfe6-cfef75c00a45 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:15:09.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=799dfaec-df12-4498-ab08-f538daacb8e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:09.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=799dfaec-df12-4498-ab08-f538daacb8e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73f86c64 2021-05-04T01:15:09.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=799dfaec-df12-4498-ab08-f538daacb8e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:09.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=799dfaec-df12-4498-ab08-f538daacb8e1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:15:11.948Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec8e309b-b126-48aa-b883-d2794bc47bab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:11.948Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec8e309b-b126-48aa-b883-d2794bc47bab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c6569b0 2021-05-04T01:15:11.948Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec8e309b-b126-48aa-b883-d2794bc47bab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:11.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec8e309b-b126-48aa-b883-d2794bc47bab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:15:14.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fb18fae-4e94-4f92-87dd-34a0ff367596 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:14.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fb18fae-4e94-4f92-87dd-34a0ff367596 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79c4a2e1 2021-05-04T01:15:14.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fb18fae-4e94-4f92-87dd-34a0ff367596 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:14.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9fb18fae-4e94-4f92-87dd-34a0ff367596 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:15:17.521Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=9fb18fae-4e94-4f92-87dd-34a0ff367596 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:15:17.521Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=9fb18fae-4e94-4f92-87dd-34a0ff367596 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:15:17.522Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9fb18fae-4e94-4f92-87dd-34a0ff367596 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:15:17.522Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9fb18fae-4e94-4f92-87dd-34a0ff367596 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:15:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:18.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:15:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:22.537Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:22.539Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:28.598Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:15:28.599Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:35.605Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:15:35.612Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:36.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:37.526Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:37.527Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:37.531Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:37.531Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f596f015-12cc-4a68-9f3c-b3eade440303 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:37.531Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f596f015-12cc-4a68-9f3c-b3eade440303 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e44a015 2021-05-04T01:15:37.531Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f596f015-12cc-4a68-9f3c-b3eade440303 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:37.531Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f596f015-12cc-4a68-9f3c-b3eade440303 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:15:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:39.616Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:15:39.617Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:40.563Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f596f015-12cc-4a68-9f3c-b3eade440303 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:15:40.563Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f596f015-12cc-4a68-9f3c-b3eade440303 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:15:40.564Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f596f015-12cc-4a68-9f3c-b3eade440303 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:15:40.564Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f596f015-12cc-4a68-9f3c-b3eade440303 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:15:40.567Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:40.569Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:40.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7cd5dfd4-4951-41a7-afa4-bd881323a39b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:40.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7cd5dfd4-4951-41a7-afa4-bd881323a39b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2900b071 2021-05-04T01:15:40.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7cd5dfd4-4951-41a7-afa4-bd881323a39b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:40.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7cd5dfd4-4951-41a7-afa4-bd881323a39b PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:15:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:42.342Z [cluster2-worker-20] ERROR c.d.driver.core.RequestHandler Unexpected error while querying cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool is shutdown at com.datastax.driver.core.HostConnectionPool.closeAsync(HostConnectionPool.java:660) at com.datastax.driver.core.SessionManager.removePool(SessionManager.java:436) at com.datastax.driver.core.SessionManager.onDown(SessionManager.java:524) at com.datastax.driver.core.Cluster$Manager.onDown(Cluster.java:2173) at com.datastax.driver.core.Cluster$Manager.access$1200(Cluster.java:1556) at com.datastax.driver.core.Cluster$Manager$5.runMayThrow(Cluster.java:2128) at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:43.599Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7cd5dfd4-4951-41a7-afa4-bd881323a39b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:15:43.599Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7cd5dfd4-4951-41a7-afa4-bd881323a39b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:15:43.599Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7cd5dfd4-4951-41a7-afa4-bd881323a39b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:15:43.599Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=7cd5dfd4-4951-41a7-afa4-bd881323a39b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:15:45.203Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:46.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01c4abac-1787-4fe9-a0bc-099daab90868 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:46.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01c4abac-1787-4fe9-a0bc-099daab90868 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@eb09ea6 2021-05-04T01:15:46.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01c4abac-1787-4fe9-a0bc-099daab90868 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:46.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01c4abac-1787-4fe9-a0bc-099daab90868 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:15:47.501Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:15:47.606Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=01c4abac-1787-4fe9-a0bc-099daab90868 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:15:47.606Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=01c4abac-1787-4fe9-a0bc-099daab90868 ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:15:48.651Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=370f1b6e-3388-485e-b414-e3764de23367 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:15:48.651Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=370f1b6e-3388-485e-b414-e3764de23367 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37027b7d 2021-05-04T01:15:48.651Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=370f1b6e-3388-485e-b414-e3764de23367 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:15:48.651Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=370f1b6e-3388-485e-b414-e3764de23367 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:15:51.657Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=370f1b6e-3388-485e-b414-e3764de23367 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:15:51.657Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=370f1b6e-3388-485e-b414-e3764de23367 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:15:51.657Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=370f1b6e-3388-485e-b414-e3764de23367 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:15:51.657Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=370f1b6e-3388-485e-b414-e3764de23367 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:15:54.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:15:54.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:15:56.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:56.700Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:15:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:16:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:16:02.702Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:16:02.703Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:16:03.201Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:16:06.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f31a151-feb6-4fa8-b03f-aa86bf249a04 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:06.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f31a151-feb6-4fa8-b03f-aa86bf249a04 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ce61e9b 2021-05-04T01:16:06.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f31a151-feb6-4fa8-b03f-aa86bf249a04 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:06.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f31a151-feb6-4fa8-b03f-aa86bf249a04 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:07.455Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9f31a151-feb6-4fa8-b03f-aa86bf249a04 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:16:07.455Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9f31a151-feb6-4fa8-b03f-aa86bf249a04 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:16:08.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36be6638-63d2-4b77-b38f-f960387f54cc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:08.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36be6638-63d2-4b77-b38f-f960387f54cc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43eb0850 2021-05-04T01:16:08.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36be6638-63d2-4b77-b38f-f960387f54cc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:08.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36be6638-63d2-4b77-b38f-f960387f54cc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:10.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=930eb008-c70f-4f4a-b2d6-c222ef1313fb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:10.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=930eb008-c70f-4f4a-b2d6-c222ef1313fb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67964ac3 2021-05-04T01:16:10.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=930eb008-c70f-4f4a-b2d6-c222ef1313fb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:10.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=930eb008-c70f-4f4a-b2d6-c222ef1313fb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:15.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:16:15.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:16:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:16:21.203Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:16:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:16:26.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfcbee2-8dbf-41e5-9dcb-c0e5f76ca929 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:26.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfcbee2-8dbf-41e5-9dcb-c0e5f76ca929 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b88826c 2021-05-04T01:16:26.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfcbee2-8dbf-41e5-9dcb-c0e5f76ca929 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:26.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfcbee2-8dbf-41e5-9dcb-c0e5f76ca929 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:26.778Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:16:28.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f371fa36-925d-4b0a-9771-08f912934ba1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:28.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f371fa36-925d-4b0a-9771-08f912934ba1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12c8ac98 2021-05-04T01:16:28.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f371fa36-925d-4b0a-9771-08f912934ba1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:28.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f371fa36-925d-4b0a-9771-08f912934ba1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:31.976Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f371fa36-925d-4b0a-9771-08f912934ba1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:16:31.977Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f371fa36-925d-4b0a-9771-08f912934ba1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:16:31.977Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f371fa36-925d-4b0a-9771-08f912934ba1 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:16:31.977Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f371fa36-925d-4b0a-9771-08f912934ba1 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:16:32.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10c6bd4d-d00e-438a-b66b-dee104cd2366 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:32.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10c6bd4d-d00e-438a-b66b-dee104cd2366 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@97a8f19 2021-05-04T01:16:32.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10c6bd4d-d00e-438a-b66b-dee104cd2366 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:32.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10c6bd4d-d00e-438a-b66b-dee104cd2366 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:32.716Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=10c6bd4d-d00e-438a-b66b-dee104cd2366 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:16:32.716Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=10c6bd4d-d00e-438a-b66b-dee104cd2366 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:16:33.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003b51f0-1672-4223-9c33-19cba329fe4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:33.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003b51f0-1672-4223-9c33-19cba329fe4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73db6eb4 2021-05-04T01:16:33.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003b51f0-1672-4223-9c33-19cba329fe4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:33.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003b51f0-1672-4223-9c33-19cba329fe4a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:35.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee82a1e6-0086-423d-8187-764b6b21f259 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:35.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee82a1e6-0086-423d-8187-764b6b21f259 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2420b03b 2021-05-04T01:16:35.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee82a1e6-0086-423d-8187-764b6b21f259 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:35.149Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee82a1e6-0086-423d-8187-764b6b21f259 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:36.255Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66efeb58-1940-486c-ab29-d9d1ab188179 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:36.255Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66efeb58-1940-486c-ab29-d9d1ab188179 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79ffe1e1 2021-05-04T01:16:36.255Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66efeb58-1940-486c-ab29-d9d1ab188179 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:36.255Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66efeb58-1940-486c-ab29-d9d1ab188179 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:37.712Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb3c49df-e981-4ce4-aef9-60ebfcc29573 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:37.712Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb3c49df-e981-4ce4-aef9-60ebfcc29573 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5dd1c05d 2021-05-04T01:16:37.712Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb3c49df-e981-4ce4-aef9-60ebfcc29573 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:37.712Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb3c49df-e981-4ce4-aef9-60ebfcc29573 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:38.950Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c92f9d67-9d69-4e2f-a6c0-654af1de225f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:38.950Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c92f9d67-9d69-4e2f-a6c0-654af1de225f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7557ab9d 2021-05-04T01:16:38.950Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c92f9d67-9d69-4e2f-a6c0-654af1de225f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:38.950Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c92f9d67-9d69-4e2f-a6c0-654af1de225f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:40.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1478a3a1-6c85-4f2c-a5cb-a7cdda2a5fbf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:40.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1478a3a1-6c85-4f2c-a5cb-a7cdda2a5fbf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54576c64 2021-05-04T01:16:40.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1478a3a1-6c85-4f2c-a5cb-a7cdda2a5fbf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:40.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1478a3a1-6c85-4f2c-a5cb-a7cdda2a5fbf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:41.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d8d8de3-0d63-4a6d-99f6-f9031d16ef60 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:41.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d8d8de3-0d63-4a6d-99f6-f9031d16ef60 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52216fe6 2021-05-04T01:16:41.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d8d8de3-0d63-4a6d-99f6-f9031d16ef60 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:41.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d8d8de3-0d63-4a6d-99f6-f9031d16ef60 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:42.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6082c0c-5f55-49b4-a58a-9c0d0f3cebda PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:42.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6082c0c-5f55-49b4-a58a-9c0d0f3cebda PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1219ed7 2021-05-04T01:16:42.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6082c0c-5f55-49b4-a58a-9c0d0f3cebda PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:42.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6082c0c-5f55-49b4-a58a-9c0d0f3cebda PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:43.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55e0cc97-f550-410b-9d46-8a7fb14acac1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:43.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55e0cc97-f550-410b-9d46-8a7fb14acac1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49e2c62d 2021-05-04T01:16:43.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55e0cc97-f550-410b-9d46-8a7fb14acac1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:43.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55e0cc97-f550-410b-9d46-8a7fb14acac1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:45.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7e3cc3-50dc-4ca5-b137-edcea6d69743 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:45.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7e3cc3-50dc-4ca5-b137-edcea6d69743 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26b9b739 2021-05-04T01:16:45.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7e3cc3-50dc-4ca5-b137-edcea6d69743 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:45.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7e3cc3-50dc-4ca5-b137-edcea6d69743 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:48.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbab4f8a-4ee1-49ed-bbb1-1f8dcad1c5d2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:48.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbab4f8a-4ee1-49ed-bbb1-1f8dcad1c5d2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fbeb257 2021-05-04T01:16:48.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbab4f8a-4ee1-49ed-bbb1-1f8dcad1c5d2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:48.398Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbab4f8a-4ee1-49ed-bbb1-1f8dcad1c5d2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:49.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=737474cb-c7c5-4942-916c-0317cb22eb3b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:49.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=737474cb-c7c5-4942-916c-0317cb22eb3b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72e44c51 2021-05-04T01:16:49.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=737474cb-c7c5-4942-916c-0317cb22eb3b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:49.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=737474cb-c7c5-4942-916c-0317cb22eb3b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:50.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cbe555d-6b12-470e-b0f9-13b6155bc1c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:50.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cbe555d-6b12-470e-b0f9-13b6155bc1c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d682f8e 2021-05-04T01:16:50.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cbe555d-6b12-470e-b0f9-13b6155bc1c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:50.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cbe555d-6b12-470e-b0f9-13b6155bc1c2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:50.899Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=715d6d9e-5b8f-4941-9d27-86a9155e70d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:50.899Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=715d6d9e-5b8f-4941-9d27-86a9155e70d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46834565 2021-05-04T01:16:50.899Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=715d6d9e-5b8f-4941-9d27-86a9155e70d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:50.899Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=715d6d9e-5b8f-4941-9d27-86a9155e70d6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:51.711Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=915cb8ca-e201-45fe-9f12-56339d8d64c0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:51.711Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=915cb8ca-e201-45fe-9f12-56339d8d64c0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c256be8 2021-05-04T01:16:51.711Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=915cb8ca-e201-45fe-9f12-56339d8d64c0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:51.711Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=915cb8ca-e201-45fe-9f12-56339d8d64c0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:52.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=364b54c7-8846-4472-8e56-5cc307f11930 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:52.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=364b54c7-8846-4472-8e56-5cc307f11930 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c7330f9 2021-05-04T01:16:52.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=364b54c7-8846-4472-8e56-5cc307f11930 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:52.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=364b54c7-8846-4472-8e56-5cc307f11930 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:54.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d541ce3-41ce-49de-a2f9-faeffbea9ed5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:54.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d541ce3-41ce-49de-a2f9-faeffbea9ed5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@708b78a6 2021-05-04T01:16:54.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d541ce3-41ce-49de-a2f9-faeffbea9ed5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:54.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d541ce3-41ce-49de-a2f9-faeffbea9ed5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:55.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98ed1491-6d0f-4a3e-bddd-d277d718caf1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:55.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98ed1491-6d0f-4a3e-bddd-d277d718caf1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c7fa3bf 2021-05-04T01:16:55.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98ed1491-6d0f-4a3e-bddd-d277d718caf1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:55.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98ed1491-6d0f-4a3e-bddd-d277d718caf1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:57.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a77dbe2-6a83-4364-bcf0-2f0e16d44d1a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:57.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a77dbe2-6a83-4364-bcf0-2f0e16d44d1a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4be2e7f3 2021-05-04T01:16:57.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a77dbe2-6a83-4364-bcf0-2f0e16d44d1a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:57.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a77dbe2-6a83-4364-bcf0-2f0e16d44d1a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:58.836Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58134f5b-d7ed-411d-9bae-6cd41024babb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:58.837Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58134f5b-d7ed-411d-9bae-6cd41024babb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e93e28e 2021-05-04T01:16:58.837Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58134f5b-d7ed-411d-9bae-6cd41024babb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:58.837Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58134f5b-d7ed-411d-9bae-6cd41024babb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:16:59.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ade68bbe-5649-42fc-914c-0992581bcfc0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:16:59.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ade68bbe-5649-42fc-914c-0992581bcfc0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6576fdb2 2021-05-04T01:16:59.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ade68bbe-5649-42fc-914c-0992581bcfc0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:16:59.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ade68bbe-5649-42fc-914c-0992581bcfc0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:00.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b902fc1-a961-41e0-bbc0-da132619dff0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:00.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b902fc1-a961-41e0-bbc0-da132619dff0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3445c8f8 2021-05-04T01:17:00.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b902fc1-a961-41e0-bbc0-da132619dff0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:00.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b902fc1-a961-41e0-bbc0-da132619dff0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:01.269Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=040a553b-e56e-4887-853c-358015844cd1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:01.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=040a553b-e56e-4887-853c-358015844cd1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68c46cfc 2021-05-04T01:17:01.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=040a553b-e56e-4887-853c-358015844cd1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:01.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=040a553b-e56e-4887-853c-358015844cd1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:02.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79a564fd-0d20-4268-bd4e-ca289ba558fb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79a564fd-0d20-4268-bd4e-ca289ba558fb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@391274f8 2021-05-04T01:17:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79a564fd-0d20-4268-bd4e-ca289ba558fb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:02.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79a564fd-0d20-4268-bd4e-ca289ba558fb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:03.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62a8f156-835d-43ae-8fb6-6639b1b06913 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:03.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62a8f156-835d-43ae-8fb6-6639b1b06913 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b5b8505 2021-05-04T01:17:03.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62a8f156-835d-43ae-8fb6-6639b1b06913 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:03.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62a8f156-835d-43ae-8fb6-6639b1b06913 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:04.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=876c8234-09ea-4e4d-a085-3f03c3c303c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:04.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=876c8234-09ea-4e4d-a085-3f03c3c303c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7143929e 2021-05-04T01:17:04.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=876c8234-09ea-4e4d-a085-3f03c3c303c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:04.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=876c8234-09ea-4e4d-a085-3f03c3c303c8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:17:06.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:17:07.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dab69e66-a550-438e-9db2-d25b5bbf617b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:07.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dab69e66-a550-438e-9db2-d25b5bbf617b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3aaba3c6 2021-05-04T01:17:07.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dab69e66-a550-438e-9db2-d25b5bbf617b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:07.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dab69e66-a550-438e-9db2-d25b5bbf617b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:07.601Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=dab69e66-a550-438e-9db2-d25b5bbf617b ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:17:07.601Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=dab69e66-a550-438e-9db2-d25b5bbf617b ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:17:08.215Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:17:08.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=969654a5-be34-45fc-a5a2-4ca666e80478 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:08.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=969654a5-be34-45fc-a5a2-4ca666e80478 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10b4b5df 2021-05-04T01:17:08.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=969654a5-be34-45fc-a5a2-4ca666e80478 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:08.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=969654a5-be34-45fc-a5a2-4ca666e80478 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:11.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76af2928-6428-4c27-9e28-9883008fa871 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:11.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76af2928-6428-4c27-9e28-9883008fa871 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27f06961 2021-05-04T01:17:11.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76af2928-6428-4c27-9e28-9883008fa871 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:11.116Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76af2928-6428-4c27-9e28-9883008fa871 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:11.400Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=76af2928-6428-4c27-9e28-9883008fa871 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:17:11.400Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=76af2928-6428-4c27-9e28-9883008fa871 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:17:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86d789c5-ff4f-4c47-9f4e-6df181a796ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86d789c5-ff4f-4c47-9f4e-6df181a796ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4acad863 2021-05-04T01:17:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86d789c5-ff4f-4c47-9f4e-6df181a796ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:14.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86d789c5-ff4f-4c47-9f4e-6df181a796ea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:18.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6271a96-6891-4115-a637-21ff94782890 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:18.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6271a96-6891-4115-a637-21ff94782890 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d98e76f 2021-05-04T01:17:18.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6271a96-6891-4115-a637-21ff94782890 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:18.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6271a96-6891-4115-a637-21ff94782890 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:20.305Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51cc2c1c-390c-4d38-96dd-8d637355b162 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:20.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51cc2c1c-390c-4d38-96dd-8d637355b162 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1837b54e 2021-05-04T01:17:20.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51cc2c1c-390c-4d38-96dd-8d637355b162 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:20.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51cc2c1c-390c-4d38-96dd-8d637355b162 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:23.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5857dfe6-794b-4732-a3c0-c06c378a8bf4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:23.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5857dfe6-794b-4732-a3c0-c06c378a8bf4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c287c48 2021-05-04T01:17:23.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5857dfe6-794b-4732-a3c0-c06c378a8bf4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:23.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5857dfe6-794b-4732-a3c0-c06c378a8bf4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:25.852Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e17bc4e-c25d-4def-bc26-d88d719fdf99 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:25.852Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e17bc4e-c25d-4def-bc26-d88d719fdf99 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20bdb26e 2021-05-04T01:17:25.852Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e17bc4e-c25d-4def-bc26-d88d719fdf99 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:25.852Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e17bc4e-c25d-4def-bc26-d88d719fdf99 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a81477b-2555-4ff4-ac2e-9f0c9c3a911d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a81477b-2555-4ff4-ac2e-9f0c9c3a911d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4af49a89 2021-05-04T01:17:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a81477b-2555-4ff4-ac2e-9f0c9c3a911d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:29.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a81477b-2555-4ff4-ac2e-9f0c9c3a911d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:32.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b3ab8c3-3620-4dc6-850f-95a26c91238d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:32.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b3ab8c3-3620-4dc6-850f-95a26c91238d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f4ccb5c 2021-05-04T01:17:32.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b3ab8c3-3620-4dc6-850f-95a26c91238d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:32.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b3ab8c3-3620-4dc6-850f-95a26c91238d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9901393d-f6b6-4855-b37b-cf51f51ff0a7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9901393d-f6b6-4855-b37b-cf51f51ff0a7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10f930c1 2021-05-04T01:17:35.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9901393d-f6b6-4855-b37b-cf51f51ff0a7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:35.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9901393d-f6b6-4855-b37b-cf51f51ff0a7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:38.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4a10981-2235-4182-9a9a-7db56ee4e045 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:38.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4a10981-2235-4182-9a9a-7db56ee4e045 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d9ade72 2021-05-04T01:17:38.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4a10981-2235-4182-9a9a-7db56ee4e045 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:38.219Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4a10981-2235-4182-9a9a-7db56ee4e045 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:41.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de63ad9e-8cb3-48e7-9c29-3d98a871ae0a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:41.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de63ad9e-8cb3-48e7-9c29-3d98a871ae0a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3354279a 2021-05-04T01:17:41.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de63ad9e-8cb3-48e7-9c29-3d98a871ae0a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:41.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de63ad9e-8cb3-48e7-9c29-3d98a871ae0a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:44.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bb3a9cc-0220-4b21-84ca-d53c0bc3c4f0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:44.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bb3a9cc-0220-4b21-84ca-d53c0bc3c4f0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a4b0c87 2021-05-04T01:17:44.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bb3a9cc-0220-4b21-84ca-d53c0bc3c4f0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:44.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bb3a9cc-0220-4b21-84ca-d53c0bc3c4f0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:46.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=672a50fd-e4ee-4ac0-9934-cc7c74b38f8d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:46.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=672a50fd-e4ee-4ac0-9934-cc7c74b38f8d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63ed5c37 2021-05-04T01:17:46.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=672a50fd-e4ee-4ac0-9934-cc7c74b38f8d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:46.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=672a50fd-e4ee-4ac0-9934-cc7c74b38f8d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:50.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f925b62-f837-477c-8684-e554bfb30d83 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:50.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f925b62-f837-477c-8684-e554bfb30d83 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@680f7776 2021-05-04T01:17:50.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f925b62-f837-477c-8684-e554bfb30d83 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:50.297Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f925b62-f837-477c-8684-e554bfb30d83 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:53.488Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9881a24-8cca-448d-8a05-d61d1619da71 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:53.489Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9881a24-8cca-448d-8a05-d61d1619da71 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a75f51a 2021-05-04T01:17:53.489Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9881a24-8cca-448d-8a05-d61d1619da71 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:53.489Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9881a24-8cca-448d-8a05-d61d1619da71 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:56.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e345259b-f072-4867-a2aa-9a0c4014aeec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:56.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e345259b-f072-4867-a2aa-9a0c4014aeec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18b8f1e7 2021-05-04T01:17:56.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e345259b-f072-4867-a2aa-9a0c4014aeec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:56.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e345259b-f072-4867-a2aa-9a0c4014aeec PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:17:59.104Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6dbf43c8-64e1-4281-a9e2-ee4ee76d63a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:17:59.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6dbf43c8-64e1-4281-a9e2-ee4ee76d63a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@11bd1a23 2021-05-04T01:17:59.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6dbf43c8-64e1-4281-a9e2-ee4ee76d63a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:17:59.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6dbf43c8-64e1-4281-a9e2-ee4ee76d63a0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:02.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d668bec3-067a-4768-afe3-5f37606d8cca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:02.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d668bec3-067a-4768-afe3-5f37606d8cca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29a6ba5 2021-05-04T01:18:02.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d668bec3-067a-4768-afe3-5f37606d8cca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:02.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d668bec3-067a-4768-afe3-5f37606d8cca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:05.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfdfba27-381f-44ca-bc49-2b10f2aac16b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:05.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfdfba27-381f-44ca-bc49-2b10f2aac16b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@571283c9 2021-05-04T01:18:05.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfdfba27-381f-44ca-bc49-2b10f2aac16b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:05.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfdfba27-381f-44ca-bc49-2b10f2aac16b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=077e7503-895f-445f-afdb-893e5d6c4205 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=077e7503-895f-445f-afdb-893e5d6c4205 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8fa6705 2021-05-04T01:18:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=077e7503-895f-445f-afdb-893e5d6c4205 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:08.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=077e7503-895f-445f-afdb-893e5d6c4205 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:11.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64a51801-1725-43af-ae86-8c2fbd0fe5a1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:11.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64a51801-1725-43af-ae86-8c2fbd0fe5a1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@716f083b 2021-05-04T01:18:11.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64a51801-1725-43af-ae86-8c2fbd0fe5a1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:11.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64a51801-1725-43af-ae86-8c2fbd0fe5a1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:14.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ad09632-234e-43c9-8870-6692fa20ceb7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:14.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ad09632-234e-43c9-8870-6692fa20ceb7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20f992f0 2021-05-04T01:18:14.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ad09632-234e-43c9-8870-6692fa20ceb7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:14.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ad09632-234e-43c9-8870-6692fa20ceb7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:17.746Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88284180-9ebc-4dd9-a103-8b5b44b889de PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:17.746Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88284180-9ebc-4dd9-a103-8b5b44b889de PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63b641ed 2021-05-04T01:18:17.746Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88284180-9ebc-4dd9-a103-8b5b44b889de PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:17.746Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88284180-9ebc-4dd9-a103-8b5b44b889de PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:20.196Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbc82486-cb50-4a30-9256-f7107fbddeab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:20.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbc82486-cb50-4a30-9256-f7107fbddeab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38afd0d1 2021-05-04T01:18:20.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbc82486-cb50-4a30-9256-f7107fbddeab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:20.197Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbc82486-cb50-4a30-9256-f7107fbddeab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e9d5b15-8b39-4baa-9380-2d4d40bf46a5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e9d5b15-8b39-4baa-9380-2d4d40bf46a5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3108292f 2021-05-04T01:18:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e9d5b15-8b39-4baa-9380-2d4d40bf46a5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:23.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e9d5b15-8b39-4baa-9380-2d4d40bf46a5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:26.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2441f8f2-8da8-4a72-a511-3ac692486fb2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:26.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2441f8f2-8da8-4a72-a511-3ac692486fb2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d0a43ec 2021-05-04T01:18:26.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2441f8f2-8da8-4a72-a511-3ac692486fb2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:26.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2441f8f2-8da8-4a72-a511-3ac692486fb2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:33.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8ab6b30-5bb5-49c3-a1c7-f8ae9a5d5269 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:33.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8ab6b30-5bb5-49c3-a1c7-f8ae9a5d5269 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ce2c889 2021-05-04T01:18:33.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8ab6b30-5bb5-49c3-a1c7-f8ae9a5d5269 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:33.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8ab6b30-5bb5-49c3-a1c7-f8ae9a5d5269 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:34.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=761f63e8-5699-4d7e-a3da-b9366d5241ef PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:34.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=761f63e8-5699-4d7e-a3da-b9366d5241ef PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a8da89c 2021-05-04T01:18:34.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=761f63e8-5699-4d7e-a3da-b9366d5241ef PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:34.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=761f63e8-5699-4d7e-a3da-b9366d5241ef PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:36.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b764d21-eb71-4e63-8dfe-bc6726d62150 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:36.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b764d21-eb71-4e63-8dfe-bc6726d62150 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a3ca1b3 2021-05-04T01:18:36.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b764d21-eb71-4e63-8dfe-bc6726d62150 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:36.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b764d21-eb71-4e63-8dfe-bc6726d62150 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:39.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112f8500-6c8d-4cc0-8e9a-ef2cba355b11 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:39.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112f8500-6c8d-4cc0-8e9a-ef2cba355b11 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35a6903b 2021-05-04T01:18:39.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112f8500-6c8d-4cc0-8e9a-ef2cba355b11 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:39.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112f8500-6c8d-4cc0-8e9a-ef2cba355b11 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33f5c2ee-4937-48c2-9054-7e589643341a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33f5c2ee-4937-48c2-9054-7e589643341a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f9aa086 2021-05-04T01:18:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33f5c2ee-4937-48c2-9054-7e589643341a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:41.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33f5c2ee-4937-48c2-9054-7e589643341a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:44.429Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8e7024f-c6f4-4a7e-a17b-d89b0819c841 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:44.429Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8e7024f-c6f4-4a7e-a17b-d89b0819c841 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8f424a1 2021-05-04T01:18:44.429Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8e7024f-c6f4-4a7e-a17b-d89b0819c841 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:44.429Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8e7024f-c6f4-4a7e-a17b-d89b0819c841 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:47.252Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be3a9103-01b6-4fe2-96f9-cd8f70947079 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:47.252Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be3a9103-01b6-4fe2-96f9-cd8f70947079 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@571a3d35 2021-05-04T01:18:47.252Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be3a9103-01b6-4fe2-96f9-cd8f70947079 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:47.252Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be3a9103-01b6-4fe2-96f9-cd8f70947079 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:50.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a700b055-8542-4168-af13-0b90686f2675 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:50.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a700b055-8542-4168-af13-0b90686f2675 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56bc58a5 2021-05-04T01:18:50.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a700b055-8542-4168-af13-0b90686f2675 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:50.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a700b055-8542-4168-af13-0b90686f2675 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b7e4780-514c-482b-82d5-dcd663e2ef8a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b7e4780-514c-482b-82d5-dcd663e2ef8a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cd84c64 2021-05-04T01:18:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b7e4780-514c-482b-82d5-dcd663e2ef8a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:53.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b7e4780-514c-482b-82d5-dcd663e2ef8a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:18:56.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a319e54f-3daa-4787-81d9-e0508343db96 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:18:56.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a319e54f-3daa-4787-81d9-e0508343db96 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@520b472 2021-05-04T01:18:56.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a319e54f-3daa-4787-81d9-e0508343db96 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:18:56.109Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a319e54f-3daa-4787-81d9-e0508343db96 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:00.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e6569ff-56e3-4923-865d-f7db5cae2150 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:00.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e6569ff-56e3-4923-865d-f7db5cae2150 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@355d276 2021-05-04T01:19:00.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e6569ff-56e3-4923-865d-f7db5cae2150 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:00.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e6569ff-56e3-4923-865d-f7db5cae2150 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:02.750Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a931f0-3304-4bc9-be33-8829a7499ef5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:02.750Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a931f0-3304-4bc9-be33-8829a7499ef5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3718cbc9 2021-05-04T01:19:02.750Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a931f0-3304-4bc9-be33-8829a7499ef5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:02.750Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a931f0-3304-4bc9-be33-8829a7499ef5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66620ca9-1216-4dbd-8b7b-c89f2bda832f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66620ca9-1216-4dbd-8b7b-c89f2bda832f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@594d0a22 2021-05-04T01:19:06.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66620ca9-1216-4dbd-8b7b-c89f2bda832f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:06.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66620ca9-1216-4dbd-8b7b-c89f2bda832f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:08.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4285f144-03cb-4586-b8ba-269d40bd9d0c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:08.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4285f144-03cb-4586-b8ba-269d40bd9d0c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59cea65d 2021-05-04T01:19:08.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4285f144-03cb-4586-b8ba-269d40bd9d0c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:08.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4285f144-03cb-4586-b8ba-269d40bd9d0c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:11.505Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4285f144-03cb-4586-b8ba-269d40bd9d0c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:19:11.506Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4285f144-03cb-4586-b8ba-269d40bd9d0c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:19:11.506Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4285f144-03cb-4586-b8ba-269d40bd9d0c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:11.506Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4285f144-03cb-4586-b8ba-269d40bd9d0c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:16.519Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:19:16.520Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:19:17.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ded90d44-e93f-4e73-b3be-b571b03f5d87 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:17.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ded90d44-e93f-4e73-b3be-b571b03f5d87 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74250970 2021-05-04T01:19:17.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ded90d44-e93f-4e73-b3be-b571b03f5d87 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:17.312Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ded90d44-e93f-4e73-b3be-b571b03f5d87 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:19:17.932Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ded90d44-e93f-4e73-b3be-b571b03f5d87 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:17.932Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ded90d44-e93f-4e73-b3be-b571b03f5d87 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:18.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac29a3d0-3cd9-472d-b2cf-a6ac6c462e04 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:18.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac29a3d0-3cd9-472d-b2cf-a6ac6c462e04 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@343fc424 2021-05-04T01:19:18.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac29a3d0-3cd9-472d-b2cf-a6ac6c462e04 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:18.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac29a3d0-3cd9-472d-b2cf-a6ac6c462e04 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:19.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ebc82a7-10cd-478c-bdab-486f0897b5e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:19.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ebc82a7-10cd-478c-bdab-486f0897b5e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18ef70b0 2021-05-04T01:19:19.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ebc82a7-10cd-478c-bdab-486f0897b5e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:19.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ebc82a7-10cd-478c-bdab-486f0897b5e0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:20.811Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ef41ce4-f199-4572-914b-29ebb6722a40 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:20.811Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ef41ce4-f199-4572-914b-29ebb6722a40 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44dd9454 2021-05-04T01:19:20.811Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ef41ce4-f199-4572-914b-29ebb6722a40 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:20.811Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ef41ce4-f199-4572-914b-29ebb6722a40 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:23.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a0d872d-ced2-488c-890c-3e14bbff43f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:23.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a0d872d-ced2-488c-890c-3e14bbff43f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5749a943 2021-05-04T01:19:23.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a0d872d-ced2-488c-890c-3e14bbff43f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:23.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a0d872d-ced2-488c-890c-3e14bbff43f5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:26.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=752c8e55-a400-4bd1-9932-b2580a16b730 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:26.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=752c8e55-a400-4bd1-9932-b2580a16b730 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d9d496a 2021-05-04T01:19:26.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=752c8e55-a400-4bd1-9932-b2580a16b730 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:26.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=752c8e55-a400-4bd1-9932-b2580a16b730 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:29.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6684616-505e-4cc9-8e37-93ad725c3422 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:29.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6684616-505e-4cc9-8e37-93ad725c3422 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e94d099 2021-05-04T01:19:29.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6684616-505e-4cc9-8e37-93ad725c3422 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:29.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6684616-505e-4cc9-8e37-93ad725c3422 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:32.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e427602-9d44-459d-bc99-b9c8a121869b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:32.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e427602-9d44-459d-bc99-b9c8a121869b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5643942b 2021-05-04T01:19:32.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e427602-9d44-459d-bc99-b9c8a121869b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:32.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e427602-9d44-459d-bc99-b9c8a121869b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:35.250Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Read timed out; blacklisting for 2 minutes 2021-05-04T01:19:35.257Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.262Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.262Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.264Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.264Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.265Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.265Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.266Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.267Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.269Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.269Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.270Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.270Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.271Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.271Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.272Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.273Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.275Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.275Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.276Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.276Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.278Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.278Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.279Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.280Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.281Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.281Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.283Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.283Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.284Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.284Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.286Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.287Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.289Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.289Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.291Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.291Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.292Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.292Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.294Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.295Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.296Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.296Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.297Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.297Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.323Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:19:35.323Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.324Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.325Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.327Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.327Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.328Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.328Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.329Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.329Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.331Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.331Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.333Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.333Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.334Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.334Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.336Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.336Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.337Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.338Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.339Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.339Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.340Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.340Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.341Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.341Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.343Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.343Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.345Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.345Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.346Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.346Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.348Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.348Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.349Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.350Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.351Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.351Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.353Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.353Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.354Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.354Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.355Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.356Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.357Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.358Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.359Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.359Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.360Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.360Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.368Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.369Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.374Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.374Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.376Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.376Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.377Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.377Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.378Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.380Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.381Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.381Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.382Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.382Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.384Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.384Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.385Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.386Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.387Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.387Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.388Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.388Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.399Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.399Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.400Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.401Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.402Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:19:35.402Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.404Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.404Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.405Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.405Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.406Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.407Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.408Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.408Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.409Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:19:35.409Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.410Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.410Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.412Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.412Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.414Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.414Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.415Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.415Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.416Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.417Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.418Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.419Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.420Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.420Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.421Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.421Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.422Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.422Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.423Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.424Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.425Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.426Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.427Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.427Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.428Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.428Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.429Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.430Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.431Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.431Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.432Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.432Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.433Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.433Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.434Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.435Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.436Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.436Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.438Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.438Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.439Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.439Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.440Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.441Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.442Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.442Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.443Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.443Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.444Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.444Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.445Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.445Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.447Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.447Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.448Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.448Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.449Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.449Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.450Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.451Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.452Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.452Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.453Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.453Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.454Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.454Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.455Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.456Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.457Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.457Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.458Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.458Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.459Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.459Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.460Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.461Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.462Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.462Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.463Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.463Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.464Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.464Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.465Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.466Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.467Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.467Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.468Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.468Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.469Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.469Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.470Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.471Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.472Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:19:35.472Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.473Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.473Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.474Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.474Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.475Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.476Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.477Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.477Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.478Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.478Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.479Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.479Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.480Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.480Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.482Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.482Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.483Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.483Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.484Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.484Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.485Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.485Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.487Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.487Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.488Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.488Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.489Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.489Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.490Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.491Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.492Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.492Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.493Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.493Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.494Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.494Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.495Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.496Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.497Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.497Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.498Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.498Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.499Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.499Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.500Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.501Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.502Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.502Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.504Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.504Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.505Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.505Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.506Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.507Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.508Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.508Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.509Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.509Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.510Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.510Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.511Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.512Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.513Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.513Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.515Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.515Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.516Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.516Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.517Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.517Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.519Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.519Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.520Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.520Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.521Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.521Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.522Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.523Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.525Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.525Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.526Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.526Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.528Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.528Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.529Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.530Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.532Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.532Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.533Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.533Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.534Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.534Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.536Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.536Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.538Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.538Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.539Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.539Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.540Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.540Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.542Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.542Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.544Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.544Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.545Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.545Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.546Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.546Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.547Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.548Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.549Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.549Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.550Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.550Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.551Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.552Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.553Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.553Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.555Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.555Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.556Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.556Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.557Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.557Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.558Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.559Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.560Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.560Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.561Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.561Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.563Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.563Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.564Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.565Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.566Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.566Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.567Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.567Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.568Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.568Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.570Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.570Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.572Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.572Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.573Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.573Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.575Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.575Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.576Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.577Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.578Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.578Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.579Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.579Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.581Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.581Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.582Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.583Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.586Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.586Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.587Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.587Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.589Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.589Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.590Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.590Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.592Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.592Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.593Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.593Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.594Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.594Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.595Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.596Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.597Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.597Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.598Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:19:35.598Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.600Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.600Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.601Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.601Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.603Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.603Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.604Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.604Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.605Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.605Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.606Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.607Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.608Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.608Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.609Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.609Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.610Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.611Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.612Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.612Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.613Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.613Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.615Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.615Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.616Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.616Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.617Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.618Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.619Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.619Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.620Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.620Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.621Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.621Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.622Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.623Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.625Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.625Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.626Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.626Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.627Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.627Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.628Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.629Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.630Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.630Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.631Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.632Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.633Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.633Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.634Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.635Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.636Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.636Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.637Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.637Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.639Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.639Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.640Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.641Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.642Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.642Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.643Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.643Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.645Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.645Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.646Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.647Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.648Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.648Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.649Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.649Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.651Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.651Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.652Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.653Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.654Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.654Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.655Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.655Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.657Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.657Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.658Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.659Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.660Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.660Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.662Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.662Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.663Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.663Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.664Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.665Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.666Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.666Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.668Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.668Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.669Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.669Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.670Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.671Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.673Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.673Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.674Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.674Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.675Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.675Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.677Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.678Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.679Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.679Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.809Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.809Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.811Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.811Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.812Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.813Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.815Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.815Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.816Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.816Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.817Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.817Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.818Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.819Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.821Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.821Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.822Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.822Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.823Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.823Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.824Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.825Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.826Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.827Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.828Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.828Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.829Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.829Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.830Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.831Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.832Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.832Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.833Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.834Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.835Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.835Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.836Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.837Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.838Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.838Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.839Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.839Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.840Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.840Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.841Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.842Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.843Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.843Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.845Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.845Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.846Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.846Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.847Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.848Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.849Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.849Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.850Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.850Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.851Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.851Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.852Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.853Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.854Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.854Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.855Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.855Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.856Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.856Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.857Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.858Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.860Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.860Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.861Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.861Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.862Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.862Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.863Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.864Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.865Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.866Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.867Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.867Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.868Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.868Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.869Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.869Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.871Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.871Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.872Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.872Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.873Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.873Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.874Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.875Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.876Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.876Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.877Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.877Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.878Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.878Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.879Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.880Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.881Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.881Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.882Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.882Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.883Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.883Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.884Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.885Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.886Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.886Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.887Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.887Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.888Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.888Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.889Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.890Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.891Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.891Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.892Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.892Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.893Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.893Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.894Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:35.899Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:35.900Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.900Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.901Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.901Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.902Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:35.902Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:35.903Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:36.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:19:36.203Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:19:36.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d0bb651-c4df-438a-a826-d32f1167baa1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:36.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d0bb651-c4df-438a-a826-d32f1167baa1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4cda3c11 2021-05-04T01:19:36.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d0bb651-c4df-438a-a826-d32f1167baa1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:36.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d0bb651-c4df-438a-a826-d32f1167baa1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:37.203Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Read timed out; blacklisting for 2 minutes 2021-05-04T01:19:37.203Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.204Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.204Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.204Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.205Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.205Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.207Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.207Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.209Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.209Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.210Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.211Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.212Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.212Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.212Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.212Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.212Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.213Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.213Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.214Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.214Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.216Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.216Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.220Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.220Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.221Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.222Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.222Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.222Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.222Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.222Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.222Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.224Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.224Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.225Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.225Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.226Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.226Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.228Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.228Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.228Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.228Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.228Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.228Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.229Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.230Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.230Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.231Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:19:37.231Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.233Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.233Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.234Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.234Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.234Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.235Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.235Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.235Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.235Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.236Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.236Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.238Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.238Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.239Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.239Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.240Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.241Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.241Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.241Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.241Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.241Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.241Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.242Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.243Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.244Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.244Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.245Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.245Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.246Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.247Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.247Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.247Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.247Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.247Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.247Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.248Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.248Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.250Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.250Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.251Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.251Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.252Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.252Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.253Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.253Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.253Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.253Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.253Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.254Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.254Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.256Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.256Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.257Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.257Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.258Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.259Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.259Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.259Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.259Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.259Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.259Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.261Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.261Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.262Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.262Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.263Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.263Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.264Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.265Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.265Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:37.265Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:19:37.265Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:19:37.265Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:19:37.265Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:19:37.267Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.267Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.268Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.268Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.269Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:37.269Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:37.270Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:37.271Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:19:37.271Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:19:38.216Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:19:38.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456606ba-71a4-4830-a018-e28b2bc60cdc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:38.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456606ba-71a4-4830-a018-e28b2bc60cdc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3bfbfaf9 2021-05-04T01:19:38.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456606ba-71a4-4830-a018-e28b2bc60cdc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:38.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456606ba-71a4-4830-a018-e28b2bc60cdc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:38.659Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=456606ba-71a4-4830-a018-e28b2bc60cdc ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:38.659Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=456606ba-71a4-4830-a018-e28b2bc60cdc ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:19:41.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc043196-4780-4b4f-a11a-414993f9372c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:41.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc043196-4780-4b4f-a11a-414993f9372c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6333e507 2021-05-04T01:19:41.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc043196-4780-4b4f-a11a-414993f9372c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:41.510Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc043196-4780-4b4f-a11a-414993f9372c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:42.013Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bc043196-4780-4b4f-a11a-414993f9372c ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:42.013Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bc043196-4780-4b4f-a11a-414993f9372c ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:44.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8016e217-f46c-446e-a49e-b0e7b0916ef1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:44.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8016e217-f46c-446e-a49e-b0e7b0916ef1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@291bc786 2021-05-04T01:19:44.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8016e217-f46c-446e-a49e-b0e7b0916ef1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:44.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8016e217-f46c-446e-a49e-b0e7b0916ef1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:19:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:19:49.293Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56e56763-abe4-4b3f-a121-f6d46dc3d804 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:49.293Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56e56763-abe4-4b3f-a121-f6d46dc3d804 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34bc5c00 2021-05-04T01:19:49.293Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56e56763-abe4-4b3f-a121-f6d46dc3d804 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:49.293Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56e56763-abe4-4b3f-a121-f6d46dc3d804 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:50.170Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:50.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:19:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:50.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:50.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:19:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:19:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:19:50.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:19:50.281Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:19:50.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e08674e8-aba7-407a-a9b4-8595f608f8c5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:50.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e08674e8-aba7-407a-a9b4-8595f608f8c5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50f4445c 2021-05-04T01:19:50.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e08674e8-aba7-407a-a9b4-8595f608f8c5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:50.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e08674e8-aba7-407a-a9b4-8595f608f8c5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:53.335Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e08674e8-aba7-407a-a9b4-8595f608f8c5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:19:53.335Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e08674e8-aba7-407a-a9b4-8595f608f8c5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:19:53.335Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e08674e8-aba7-407a-a9b4-8595f608f8c5 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:53.336Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=e08674e8-aba7-407a-a9b4-8595f608f8c5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:19:54.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5d23fa8-d019-4118-8094-18a7fc89c158 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:54.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5d23fa8-d019-4118-8094-18a7fc89c158 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c138898 2021-05-04T01:19:54.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5d23fa8-d019-4118-8094-18a7fc89c158 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:54.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5d23fa8-d019-4118-8094-18a7fc89c158 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:55.211Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a5d23fa8-d019-4118-8094-18a7fc89c158 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:55.211Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a5d23fa8-d019-4118-8094-18a7fc89c158 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:19:56.448Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=373191bb-4fd0-4978-95ed-82fa048580f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:56.448Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=373191bb-4fd0-4978-95ed-82fa048580f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@dc9565f 2021-05-04T01:19:56.448Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=373191bb-4fd0-4978-95ed-82fa048580f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:56.448Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=373191bb-4fd0-4978-95ed-82fa048580f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:19:59.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b8520d-c739-4fb3-93a1-6f69f2f329c0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:19:59.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b8520d-c739-4fb3-93a1-6f69f2f329c0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fed251e 2021-05-04T01:19:59.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b8520d-c739-4fb3-93a1-6f69f2f329c0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:19:59.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b8520d-c739-4fb3-93a1-6f69f2f329c0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:02.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cb99c9b-704a-488e-83d0-fc38fcf732bc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:02.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cb99c9b-704a-488e-83d0-fc38fcf732bc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f8a2533 2021-05-04T01:20:02.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cb99c9b-704a-488e-83d0-fc38fcf732bc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:02.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cb99c9b-704a-488e-83d0-fc38fcf732bc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:20:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:05.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:20:05.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:20:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:05.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:20:05.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=700debc2-1d4e-41d8-af3e-f667905a1687 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:05.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=700debc2-1d4e-41d8-af3e-f667905a1687 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e6301bc 2021-05-04T01:20:05.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=700debc2-1d4e-41d8-af3e-f667905a1687 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:05.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=700debc2-1d4e-41d8-af3e-f667905a1687 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01k6y80e57lhid1hmbclj54zi0l291 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01dqwovd9h07argish43ccwbkw304 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ep70xsq6nvsb1942vxgfmg4wm300 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0jl88ohl4u87u13rhi9dw817wh301 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0cz4h017hc25j1vp1g8iop0y2l302 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01x1s5ikfq1htohk8uwqod4v0r306 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01j7q9qbsdysxsvftf4wjr6bk5294 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node08dkb4aj2ysf5dosyru0jfi9297 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0vlbqvapvz39etb9v8ypciiuv307 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0bgm6zbdne08w15ll2l344hml1299 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0eke0vxl2182h6fzqg3qdfk6a298 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01kkyribc7n42cxx7r6z7lkgqx305 2021-05-04T01:20:08.077Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0wsbtod65q3721wmfeshcefpl8303 2021-05-04T01:20:08.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e36442c9-d2cd-494f-a963-65ef24ea905e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:08.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e36442c9-d2cd-494f-a963-65ef24ea905e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@495fd581 2021-05-04T01:20:08.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e36442c9-d2cd-494f-a963-65ef24ea905e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:08.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e36442c9-d2cd-494f-a963-65ef24ea905e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:20:15.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:20:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:20.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:20.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:20.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:20:20.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:20:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:20:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:20.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:20.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:20.186Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:20:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:25.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d878e3d-dbdd-430b-8670-2f2e01ec2ee5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:25.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d878e3d-dbdd-430b-8670-2f2e01ec2ee5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f0ccc61 2021-05-04T01:20:25.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d878e3d-dbdd-430b-8670-2f2e01ec2ee5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:25.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d878e3d-dbdd-430b-8670-2f2e01ec2ee5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:25.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d76320a-0fc1-407f-95ce-cd220ad6a195 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:25.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d76320a-0fc1-407f-95ce-cd220ad6a195 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4131c90e 2021-05-04T01:20:25.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d76320a-0fc1-407f-95ce-cd220ad6a195 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:25.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d76320a-0fc1-407f-95ce-cd220ad6a195 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:25.841Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2d76320a-0fc1-407f-95ce-cd220ad6a195 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:20:25.842Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=2d76320a-0fc1-407f-95ce-cd220ad6a195 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:20:26.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1806dfbc-68cc-4c20-8dd7-1bf1c21837ce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:26.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1806dfbc-68cc-4c20-8dd7-1bf1c21837ce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57494e75 2021-05-04T01:20:26.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1806dfbc-68cc-4c20-8dd7-1bf1c21837ce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:26.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1806dfbc-68cc-4c20-8dd7-1bf1c21837ce PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:26.245Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:20:26.454Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe1c6e1a-f6e8-44a0-bb2e-cee161b4bd4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:26.454Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe1c6e1a-f6e8-44a0-bb2e-cee161b4bd4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@600d6ac0 2021-05-04T01:20:26.455Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe1c6e1a-f6e8-44a0-bb2e-cee161b4bd4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:26.455Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe1c6e1a-f6e8-44a0-bb2e-cee161b4bd4e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=472875ce-aae7-4b10-89f9-fd59423d9d33 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=472875ce-aae7-4b10-89f9-fd59423d9d33 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59dbdd64 2021-05-04T01:20:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=472875ce-aae7-4b10-89f9-fd59423d9d33 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=472875ce-aae7-4b10-89f9-fd59423d9d33 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:29.871Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=472875ce-aae7-4b10-89f9-fd59423d9d33 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:20:29.872Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=472875ce-aae7-4b10-89f9-fd59423d9d33 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:20:29.872Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=472875ce-aae7-4b10-89f9-fd59423d9d33 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:20:29.872Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=472875ce-aae7-4b10-89f9-fd59423d9d33 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:20:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:20:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:20:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:35.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:20:35.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:20:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:35.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:20:36.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:20:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:20:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:20:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:20:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:20:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:20:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:20:38.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4e7e30-0c45-4876-b0f1-07b7cfad2dae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:38.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4e7e30-0c45-4876-b0f1-07b7cfad2dae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@166d16e8 2021-05-04T01:20:38.477Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4e7e30-0c45-4876-b0f1-07b7cfad2dae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:38.477Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4e7e30-0c45-4876-b0f1-07b7cfad2dae PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:39.081Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:20:40.014Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3e4e7e30-0c45-4876-b0f1-07b7cfad2dae ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:20:40.014Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3e4e7e30-0c45-4876-b0f1-07b7cfad2dae ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:20:40.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa515c8-b863-4896-946f-7c8889daa0c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:40.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa515c8-b863-4896-946f-7c8889daa0c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9af875 2021-05-04T01:20:40.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa515c8-b863-4896-946f-7c8889daa0c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:40.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa515c8-b863-4896-946f-7c8889daa0c1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:20:42.205Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:42.205Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:20:43.904Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=aaa515c8-b863-4896-946f-7c8889daa0c1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:20:43.904Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=aaa515c8-b863-4896-946f-7c8889daa0c1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:20:43.905Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=aaa515c8-b863-4896-946f-7c8889daa0c1 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:20:43.905Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=aaa515c8-b863-4896-946f-7c8889daa0c1 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:20:43.909Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:43.910Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:43.912Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:43.912Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4f26bc-02ab-43bb-b259-135af4bb9a26 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:43.912Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4f26bc-02ab-43bb-b259-135af4bb9a26 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72d35ff 2021-05-04T01:20:43.912Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4f26bc-02ab-43bb-b259-135af4bb9a26 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:43.912Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4f26bc-02ab-43bb-b259-135af4bb9a26 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:20:44.915Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:20:44.916Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:45.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:46.921Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:20:46.922Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:46.961Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0b4f26bc-02ab-43bb-b259-135af4bb9a26 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:20:46.961Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0b4f26bc-02ab-43bb-b259-135af4bb9a26 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:20:46.962Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0b4f26bc-02ab-43bb-b259-135af4bb9a26 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:20:46.962Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=0b4f26bc-02ab-43bb-b259-135af4bb9a26 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:20:46.965Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:46.968Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:46.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24ff349f-3510-4e39-a853-70e63e67efc4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:46.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24ff349f-3510-4e39-a853-70e63e67efc4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@245b415b 2021-05-04T01:20:46.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24ff349f-3510-4e39-a853-70e63e67efc4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:46.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24ff349f-3510-4e39-a853-70e63e67efc4 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:20:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:49.996Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=24ff349f-3510-4e39-a853-70e63e67efc4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:20:49.997Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=24ff349f-3510-4e39-a853-70e63e67efc4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:20:50.000Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:50.002Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:50.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14bcaa1c-93ed-4a38-a51e-ba2420fbaa3c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:50.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14bcaa1c-93ed-4a38-a51e-ba2420fbaa3c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1db91dbd 2021-05-04T01:20:50.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14bcaa1c-93ed-4a38-a51e-ba2420fbaa3c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:50.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14bcaa1c-93ed-4a38-a51e-ba2420fbaa3c PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:20:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:20:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:20:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:20:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:50.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:20:50.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:20:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:50.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:20:50.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:20:50.187Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:20:50.928Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:20:50.931Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:53.035Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=14bcaa1c-93ed-4a38-a51e-ba2420fbaa3c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:20:53.036Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=14bcaa1c-93ed-4a38-a51e-ba2420fbaa3c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:20:53.100Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:53.102Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:53.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3f223a7-b540-4745-8dae-d894543e2b3f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:53.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3f223a7-b540-4745-8dae-d894543e2b3f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57e61edc 2021-05-04T01:20:53.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3f223a7-b540-4745-8dae-d894543e2b3f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:53.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3f223a7-b540-4745-8dae-d894543e2b3f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:20:54.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:56.139Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a3f223a7-b540-4745-8dae-d894543e2b3f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:20:56.140Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a3f223a7-b540-4745-8dae-d894543e2b3f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:20:56.200Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:56.202Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a072cf4d-6a26-4fe5-a150-2c72261990e4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a072cf4d-6a26-4fe5-a150-2c72261990e4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b0c62a8 2021-05-04T01:20:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a072cf4d-6a26-4fe5-a150-2c72261990e4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:56.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a072cf4d-6a26-4fe5-a150-2c72261990e4 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:20:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:20:58.998Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:20:59.000Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 16000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:59.243Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a072cf4d-6a26-4fe5-a150-2c72261990e4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:20:59.243Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a072cf4d-6a26-4fe5-a150-2c72261990e4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:20:59.246Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:59.248Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:20:59.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=162b7ae3-0946-4d35-ba2f-c83f3c1ad24b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:20:59.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=162b7ae3-0946-4d35-ba2f-c83f3c1ad24b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5bccbbb0 2021-05-04T01:20:59.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=162b7ae3-0946-4d35-ba2f-c83f3c1ad24b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:20:59.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=162b7ae3-0946-4d35-ba2f-c83f3c1ad24b PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:21:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:02.287Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=162b7ae3-0946-4d35-ba2f-c83f3c1ad24b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:21:02.288Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=162b7ae3-0946-4d35-ba2f-c83f3c1ad24b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:21:02.300Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:02.304Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:02.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ca9c09d-0782-4f1c-acab-5fd9d7681c2d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:21:02.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ca9c09d-0782-4f1c-acab-5fd9d7681c2d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3915ade 2021-05-04T01:21:02.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ca9c09d-0782-4f1c-acab-5fd9d7681c2d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:21:02.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ca9c09d-0782-4f1c-acab-5fd9d7681c2d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:21:03.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:21:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:05.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:21:05.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:21:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:05.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:21:05.332Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6ca9c09d-0782-4f1c-acab-5fd9d7681c2d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:21:05.332Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6ca9c09d-0782-4f1c-acab-5fd9d7681c2d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:21:05.334Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:05.336Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:05.336Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0a7b25f-6ef2-4363-bfde-5f797851def5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:21:05.336Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0a7b25f-6ef2-4363-bfde-5f797851def5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5bf3bd63 2021-05-04T01:21:05.336Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0a7b25f-6ef2-4363-bfde-5f797851def5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:21:05.336Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0a7b25f-6ef2-4363-bfde-5f797851def5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:21:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:08.361Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f0a7b25f-6ef2-4363-bfde-5f797851def5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:21:08.361Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f0a7b25f-6ef2-4363-bfde-5f797851def5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:21:08.365Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:09.275Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:09.275Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d36e9e11-3112-4dc9-80ee-82f9fc4c5955 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:21:09.275Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d36e9e11-3112-4dc9-80ee-82f9fc4c5955 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a04b044 2021-05-04T01:21:09.275Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d36e9e11-3112-4dc9-80ee-82f9fc4c5955 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:21:09.275Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d36e9e11-3112-4dc9-80ee-82f9fc4c5955 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:21:10.360Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:85) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:21:10.360Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:832) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:21:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:12.314Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d36e9e11-3112-4dc9-80ee-82f9fc4c5955 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:21:12.314Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d36e9e11-3112-4dc9-80ee-82f9fc4c5955 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:21:12.315Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d36e9e11-3112-4dc9-80ee-82f9fc4c5955 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:21:12.315Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=d36e9e11-3112-4dc9-80ee-82f9fc4c5955 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:21:15.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:16.553Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:85) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:21:17.148Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:832) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:21:18.082Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed16ef13-db6f-4739-bbdb-59e4676f331a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:21:18.082Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed16ef13-db6f-4739-bbdb-59e4676f331a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d5d18e2 2021-05-04T01:21:18.082Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed16ef13-db6f-4739-bbdb-59e4676f331a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:21:18.082Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed16ef13-db6f-4739-bbdb-59e4676f331a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:21:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:21:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:20.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:21:20.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:21:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:20.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:20.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:20.187Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:21:21.115Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=ed16ef13-db6f-4739-bbdb-59e4676f331a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:21:21.115Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=ed16ef13-db6f-4739-bbdb-59e4676f331a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:21:21.203Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:24.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:25.670Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:832) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:21:25.670Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:21:09.822109Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-05-04T01:21:26.128Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a7f6158-a4de-4978-bc78-416915db6f97 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:21:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a7f6158-a4de-4978-bc78-416915db6f97 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53e5d158 2021-05-04T01:21:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a7f6158-a4de-4978-bc78-416915db6f97 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:21:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a7f6158-a4de-4978-bc78-416915db6f97 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:21:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:29.212Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:21:29.993Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4a7f6158-a4de-4978-bc78-416915db6f97 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:21:29.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4a7f6158-a4de-4978-bc78-416915db6f97 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:21:32.133Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:21:32.134Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:21:33.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3870bc9a-dd9a-4fc6-ae78-bf3b9a862d00 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:21:33.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3870bc9a-dd9a-4fc6-ae78-bf3b9a862d00 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@600d0428 2021-05-04T01:21:33.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3870bc9a-dd9a-4fc6-ae78-bf3b9a862d00 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:21:33.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3870bc9a-dd9a-4fc6-ae78-bf3b9a862d00 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:21:34.143Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:21:34.147Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:21:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:35.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:21:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:21:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:35.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:21:35.307Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:21:36.475Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=3870bc9a-dd9a-4fc6-ae78-bf3b9a862d00 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:21:36.475Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=3870bc9a-dd9a-4fc6-ae78-bf3b9a862d00 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:21:36.476Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3870bc9a-dd9a-4fc6-ae78-bf3b9a862d00 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:21:36.476Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3870bc9a-dd9a-4fc6-ae78-bf3b9a862d00 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:21:36.502Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:21:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:21:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:21:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:21:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:21:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:21:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:21:37.301Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:37.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c666965a-255f-469e-b9ae-a1edd6dc25d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:21:37.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c666965a-255f-469e-b9ae-a1edd6dc25d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67e975ba 2021-05-04T01:21:37.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c666965a-255f-469e-b9ae-a1edd6dc25d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:21:37.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c666965a-255f-469e-b9ae-a1edd6dc25d7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:21:38.151Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:21:38.152Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:40.331Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c666965a-255f-469e-b9ae-a1edd6dc25d7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:21:40.331Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c666965a-255f-469e-b9ae-a1edd6dc25d7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:21:40.335Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:21:45.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:46.157Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:21:46.158Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 16000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:21:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:21:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:50.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:21:50.186Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:21:50.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:50.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:50.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:50.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:50.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:21:50.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:21:50.191Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:21:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:54.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:21:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:22:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:22:00.427Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:00.427Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:00.431Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:00.431Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5555fa2-ab80-43b2-9c51-f0b5bf5b49b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:00.431Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5555fa2-ab80-43b2-9c51-f0b5bf5b49b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e3d1842 2021-05-04T01:22:00.431Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5555fa2-ab80-43b2-9c51-f0b5bf5b49b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:00.431Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5555fa2-ab80-43b2-9c51-f0b5bf5b49b8 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:22:02.165Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:22:02.166Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 32000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:03.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:22:03.472Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=b5555fa2-ab80-43b2-9c51-f0b5bf5b49b8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:03.473Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=b5555fa2-ab80-43b2-9c51-f0b5bf5b49b8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:03.473Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b5555fa2-ab80-43b2-9c51-f0b5bf5b49b8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:22:03.473Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=b5555fa2-ab80-43b2-9c51-f0b5bf5b49b8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:22:03.500Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:03.502Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cb0f88c-8e13-4b32-b833-6a84f3c74e34 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cb0f88c-8e13-4b32-b833-6a84f3c74e34 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30293756 2021-05-04T01:22:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cb0f88c-8e13-4b32-b833-6a84f3c74e34 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cb0f88c-8e13-4b32-b833-6a84f3c74e34 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:22:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:22:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:05.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:22:05.184Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:22:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:05.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:05.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:05.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:05.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:05.190Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:22:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:22:06.540Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=3cb0f88c-8e13-4b32-b833-6a84f3c74e34 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:06.541Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=3cb0f88c-8e13-4b32-b833-6a84f3c74e34 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:06.541Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3cb0f88c-8e13-4b32-b833-6a84f3c74e34 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:22:06.541Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=3cb0f88c-8e13-4b32-b833-6a84f3c74e34 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:22:06.544Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:06.545Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:06.545Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e0fb3f2-caa0-496d-8a69-bdbe398fe2fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:06.545Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e0fb3f2-caa0-496d-8a69-bdbe398fe2fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b330fc9 2021-05-04T01:22:06.545Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e0fb3f2-caa0-496d-8a69-bdbe398fe2fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:06.546Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e0fb3f2-caa0-496d-8a69-bdbe398fe2fe PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:22:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:22:09.580Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=3e0fb3f2-caa0-496d-8a69-bdbe398fe2fe ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:09.581Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=3e0fb3f2-caa0-496d-8a69-bdbe398fe2fe ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:09.583Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:09.585Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:09.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d2983e9-03c5-44db-b0bb-901f93264eb4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:09.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d2983e9-03c5-44db-b0bb-901f93264eb4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70242bdb 2021-05-04T01:22:09.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d2983e9-03c5-44db-b0bb-901f93264eb4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:09.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d2983e9-03c5-44db-b0bb-901f93264eb4 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:22:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:22:12.623Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=1d2983e9-03c5-44db-b0bb-901f93264eb4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:12.623Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=1d2983e9-03c5-44db-b0bb-901f93264eb4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:12.626Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:12.628Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:12.628Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=478d76b6-5d7f-4856-80a0-d1fb9975b82e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:12.628Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=478d76b6-5d7f-4856-80a0-d1fb9975b82e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9033029 2021-05-04T01:22:12.628Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=478d76b6-5d7f-4856-80a0-d1fb9975b82e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:12.628Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=478d76b6-5d7f-4856-80a0-d1fb9975b82e PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:22:15.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:22:15.665Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=478d76b6-5d7f-4856-80a0-d1fb9975b82e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:15.665Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=478d76b6-5d7f-4856-80a0-d1fb9975b82e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:15.669Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:15.707Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:15.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d9bac9b-16b8-4c57-b4cb-1271584c156f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:15.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d9bac9b-16b8-4c57-b4cb-1271584c156f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@206f2f1c 2021-05-04T01:22:15.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d9bac9b-16b8-4c57-b4cb-1271584c156f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:15.707Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d9bac9b-16b8-4c57-b4cb-1271584c156f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:22:16.075Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:832) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-05-04T01:22:16.076Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:22:17.806Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:22:18.752Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2d9bac9b-16b8-4c57-b4cb-1271584c156f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:18.752Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2d9bac9b-16b8-4c57-b4cb-1271584c156f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:19.844Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68c45332-6d79-453b-8163-b37fa66e21ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:19.844Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68c45332-6d79-453b-8163-b37fa66e21ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@530c0016 2021-05-04T01:22:19.845Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68c45332-6d79-453b-8163-b37fa66e21ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:19.845Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68c45332-6d79-453b-8163-b37fa66e21ac PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:22:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:22:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:20.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:22:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:22:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:20.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:22:22.881Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=68c45332-6d79-453b-8163-b37fa66e21ac ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:22.882Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=68c45332-6d79-453b-8163-b37fa66e21ac ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:22.882Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=68c45332-6d79-453b-8163-b37fa66e21ac ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:22:22.882Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=68c45332-6d79-453b-8163-b37fa66e21ac ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:22:27.902Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:34.175Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:22:34.179Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 64000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:22:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:35.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:22:35.182Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:22:35.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:35.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:35.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:35.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:35.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:35.190Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:22:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:22:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:22:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:22:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:22:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:37.114Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:22:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:22:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:22:37.665Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:22:37.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f74c736-8bba-42b9-b071-a6a6f80306bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:37.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f74c736-8bba-42b9-b071-a6a6f80306bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e921e8b 2021-05-04T01:22:37.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f74c736-8bba-42b9-b071-a6a6f80306bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:37.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f74c736-8bba-42b9-b071-a6a6f80306bd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:22:40.712Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8f74c736-8bba-42b9-b071-a6a6f80306bd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:40.712Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8f74c736-8bba-42b9-b071-a6a6f80306bd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:42.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86c04b2c-cd01-4946-91c2-00cee62dd3ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:42.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86c04b2c-cd01-4946-91c2-00cee62dd3ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5916e95c 2021-05-04T01:22:42.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86c04b2c-cd01-4946-91c2-00cee62dd3ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:42.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86c04b2c-cd01-4946-91c2-00cee62dd3ba PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:22:45.087Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=86c04b2c-cd01-4946-91c2-00cee62dd3ba ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:45.088Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=86c04b2c-cd01-4946-91c2-00cee62dd3ba ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:45.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1d493a0-8eac-43e3-a73e-8ab794df2666 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:45.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1d493a0-8eac-43e3-a73e-8ab794df2666 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76b6e2cd 2021-05-04T01:22:45.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1d493a0-8eac-43e3-a73e-8ab794df2666 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:45.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1d493a0-8eac-43e3-a73e-8ab794df2666 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:22:48.738Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=b1d493a0-8eac-43e3-a73e-8ab794df2666 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:22:48.738Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=b1d493a0-8eac-43e3-a73e-8ab794df2666 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:22:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:22:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:50.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:22:50.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:22:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:22:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:22:50.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:22:50.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a121278-7f18-43b2-a801-dbd439d76151 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:50.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a121278-7f18-43b2-a801-dbd439d76151 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38487f0e 2021-05-04T01:22:50.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a121278-7f18-43b2-a801-dbd439d76151 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:50.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a121278-7f18-43b2-a801-dbd439d76151 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:22:51.754Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9a121278-7f18-43b2-a801-dbd439d76151 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:22:51.755Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9a121278-7f18-43b2-a801-dbd439d76151 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:22:52.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03f407d7-2e05-4518-984a-01f4346a54a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:52.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03f407d7-2e05-4518-984a-01f4346a54a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37d12758 2021-05-04T01:22:52.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03f407d7-2e05-4518-984a-01f4346a54a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:52.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03f407d7-2e05-4518-984a-01f4346a54a2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:22:53.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2801a5ff-1391-4e55-991d-a2a0aaf16bcd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:53.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2801a5ff-1391-4e55-991d-a2a0aaf16bcd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30a85f36 2021-05-04T01:22:53.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2801a5ff-1391-4e55-991d-a2a0aaf16bcd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:53.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2801a5ff-1391-4e55-991d-a2a0aaf16bcd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:22:55.688Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7f274a7-c8db-4c46-b6b4-a90a8c593f4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:55.688Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7f274a7-c8db-4c46-b6b4-a90a8c593f4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b63f622 2021-05-04T01:22:55.688Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7f274a7-c8db-4c46-b6b4-a90a8c593f4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:55.688Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7f274a7-c8db-4c46-b6b4-a90a8c593f4e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:22:56.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aba3d539-947c-4704-88de-67b1046e626f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:56.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aba3d539-947c-4704-88de-67b1046e626f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d3b0acc 2021-05-04T01:22:56.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aba3d539-947c-4704-88de-67b1046e626f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:56.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aba3d539-947c-4704-88de-67b1046e626f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:22:58.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=60952fd9-65a1-465b-9e0e-d88769cba6ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:22:58.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=60952fd9-65a1-465b-9e0e-d88769cba6ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ee01d4 2021-05-04T01:22:58.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=60952fd9-65a1-465b-9e0e-d88769cba6ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:22:58.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=60952fd9-65a1-465b-9e0e-d88769cba6ff PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:00.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbd7e8d4-76b9-4067-93ff-79af37d66f98 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:00.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbd7e8d4-76b9-4067-93ff-79af37d66f98 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@714dbf70 2021-05-04T01:23:00.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbd7e8d4-76b9-4067-93ff-79af37d66f98 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:00.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbd7e8d4-76b9-4067-93ff-79af37d66f98 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:01.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5554eaef-7ab2-4cc3-80a6-fa64401fd8f2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:01.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5554eaef-7ab2-4cc3-80a6-fa64401fd8f2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@254b959d 2021-05-04T01:23:01.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5554eaef-7ab2-4cc3-80a6-fa64401fd8f2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:01.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5554eaef-7ab2-4cc3-80a6-fa64401fd8f2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:04.376Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5554eaef-7ab2-4cc3-80a6-fa64401fd8f2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:23:04.376Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5554eaef-7ab2-4cc3-80a6-fa64401fd8f2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:23:04.377Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5554eaef-7ab2-4cc3-80a6-fa64401fd8f2 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:23:04.377Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5554eaef-7ab2-4cc3-80a6-fa64401fd8f2 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:23:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:23:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:05.174Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:23:05.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:23:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:05.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:23:06.946Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55002287-acfe-48a2-9e5e-422175430d1f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:06.947Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55002287-acfe-48a2-9e5e-422175430d1f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40c39692 2021-05-04T01:23:06.947Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55002287-acfe-48a2-9e5e-422175430d1f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:06.947Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55002287-acfe-48a2-9e5e-422175430d1f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:07.099Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=55002287-acfe-48a2-9e5e-422175430d1f ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:23:07.099Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=55002287-acfe-48a2-9e5e-422175430d1f ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:23:07.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ed9320e-b2fd-487b-adb4-d836da758575 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:07.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ed9320e-b2fd-487b-adb4-d836da758575 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f7a50da 2021-05-04T01:23:07.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ed9320e-b2fd-487b-adb4-d836da758575 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:07.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ed9320e-b2fd-487b-adb4-d836da758575 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:07.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e86a0c2-d4ce-479e-9bac-efd0e5fd2384 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:07.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e86a0c2-d4ce-479e-9bac-efd0e5fd2384 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56e4cfd5 2021-05-04T01:23:07.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e86a0c2-d4ce-479e-9bac-efd0e5fd2384 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:07.989Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e86a0c2-d4ce-479e-9bac-efd0e5fd2384 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:09.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590b12c6-a02e-4ff9-b435-5bf8682cc52d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:09.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590b12c6-a02e-4ff9-b435-5bf8682cc52d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a4fb49d 2021-05-04T01:23:09.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590b12c6-a02e-4ff9-b435-5bf8682cc52d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:09.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590b12c6-a02e-4ff9-b435-5bf8682cc52d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:11.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e87bfbb1-fd15-4106-aea3-512399723845 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:11.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e87bfbb1-fd15-4106-aea3-512399723845 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f55bd1c 2021-05-04T01:23:11.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e87bfbb1-fd15-4106-aea3-512399723845 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:11.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e87bfbb1-fd15-4106-aea3-512399723845 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:11.354Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6175e2d-245a-4f82-825d-0c7febae15e3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:11.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6175e2d-245a-4f82-825d-0c7febae15e3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4254d6ac 2021-05-04T01:23:11.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6175e2d-245a-4f82-825d-0c7febae15e3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:11.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6175e2d-245a-4f82-825d-0c7febae15e3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:11.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69955812-2c9a-4e48-bd30-7a0db2c53321 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:11.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69955812-2c9a-4e48-bd30-7a0db2c53321 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32be2354 2021-05-04T01:23:11.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69955812-2c9a-4e48-bd30-7a0db2c53321 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:11.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69955812-2c9a-4e48-bd30-7a0db2c53321 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:13.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5160260-ee16-49d0-9634-abcbc0d1f723 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:13.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5160260-ee16-49d0-9634-abcbc0d1f723 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45a292c2 2021-05-04T01:23:13.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5160260-ee16-49d0-9634-abcbc0d1f723 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:13.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5160260-ee16-49d0-9634-abcbc0d1f723 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:14.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0357128-b054-4e6c-b29a-1ffaf491c31d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:14.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0357128-b054-4e6c-b29a-1ffaf491c31d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@570d6a37 2021-05-04T01:23:14.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0357128-b054-4e6c-b29a-1ffaf491c31d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:14.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0357128-b054-4e6c-b29a-1ffaf491c31d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:16.581Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc2dd3b-792d-44ef-ba2f-972c9d8f2209 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:16.581Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc2dd3b-792d-44ef-ba2f-972c9d8f2209 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@330827cc 2021-05-04T01:23:16.581Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc2dd3b-792d-44ef-ba2f-972c9d8f2209 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:16.581Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc2dd3b-792d-44ef-ba2f-972c9d8f2209 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:17.905Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=460b1733-98db-4880-b25b-99f11c740b6c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:17.905Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=460b1733-98db-4880-b25b-99f11c740b6c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@706f067a 2021-05-04T01:23:17.905Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=460b1733-98db-4880-b25b-99f11c740b6c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:17.905Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=460b1733-98db-4880-b25b-99f11c740b6c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:18.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f939f8da-1d7a-49f5-b454-fc25dafb8cc0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:18.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f939f8da-1d7a-49f5-b454-fc25dafb8cc0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@218aae37 2021-05-04T01:23:18.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f939f8da-1d7a-49f5-b454-fc25dafb8cc0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:18.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f939f8da-1d7a-49f5-b454-fc25dafb8cc0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:19.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0df584f-0455-4e97-9c7a-1193155dff42 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:19.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0df584f-0455-4e97-9c7a-1193155dff42 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d6a24e3 2021-05-04T01:23:19.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0df584f-0455-4e97-9c7a-1193155dff42 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:19.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0df584f-0455-4e97-9c7a-1193155dff42 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:23:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:20.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:23:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:23:20.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:20.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:20.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:23:20.976Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1e8046a-fec2-4590-8905-31e8d0fba11f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:20.976Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1e8046a-fec2-4590-8905-31e8d0fba11f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f2e46c5 2021-05-04T01:23:20.976Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1e8046a-fec2-4590-8905-31e8d0fba11f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:20.977Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1e8046a-fec2-4590-8905-31e8d0fba11f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:22.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05ab5f8c-a0c7-4f1a-a144-e857386cf4fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:22.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05ab5f8c-a0c7-4f1a-a144-e857386cf4fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67fad18c 2021-05-04T01:23:22.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05ab5f8c-a0c7-4f1a-a144-e857386cf4fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:22.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05ab5f8c-a0c7-4f1a-a144-e857386cf4fa PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:24.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:23:24.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:23:26.346Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:23:27.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a16050f4-b907-4411-a48e-321eb3e7f57d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:27.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a16050f4-b907-4411-a48e-321eb3e7f57d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13753052 2021-05-04T01:23:27.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a16050f4-b907-4411-a48e-321eb3e7f57d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:27.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a16050f4-b907-4411-a48e-321eb3e7f57d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:28.077Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=209be622-ea33-4fc3-9435-fa026de9ce77 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:28.078Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=209be622-ea33-4fc3-9435-fa026de9ce77 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ff443e3 2021-05-04T01:23:28.078Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=209be622-ea33-4fc3-9435-fa026de9ce77 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:28.078Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=209be622-ea33-4fc3-9435-fa026de9ce77 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:30.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47ccb872-27ec-4753-8991-e8d0685dfb6f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:30.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47ccb872-27ec-4753-8991-e8d0685dfb6f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42712eb6 2021-05-04T01:23:30.505Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47ccb872-27ec-4753-8991-e8d0685dfb6f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:30.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47ccb872-27ec-4753-8991-e8d0685dfb6f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:31.388Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d9734b-b0c8-4924-a685-d93a868f2515 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:31.388Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d9734b-b0c8-4924-a685-d93a868f2515 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73b6e1c5 2021-05-04T01:23:31.388Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d9734b-b0c8-4924-a685-d93a868f2515 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:31.388Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d9734b-b0c8-4924-a685-d93a868f2515 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:32.653Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ad20f63-75af-407c-99d4-fd578b5620cc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:32.653Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ad20f63-75af-407c-99d4-fd578b5620cc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49fba841 2021-05-04T01:23:32.653Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ad20f63-75af-407c-99d4-fd578b5620cc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:32.653Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ad20f63-75af-407c-99d4-fd578b5620cc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:34.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:34.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@346d11c6 2021-05-04T01:23:34.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:34.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:23:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:23:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:35.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:23:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:23:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:35.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:23:36.210Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:23:36.211Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:23:36.407Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 PartnerName=SDC.BE I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://sdc-onboarding-be.onap:8445: The target server failed to respond 2021-05-04T01:23:36.407Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 PartnerName=SDC.BE Retrying request to {s}->https://sdc-onboarding-be.onap:8445 2021-05-04T01:23:36.412Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:23:36.413Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:23:36.413Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:23:36.413Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2ed5cc3d-3630-4e0c-abdd-1d1b5143f067 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:23:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:23:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:23:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:23:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:23:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:23:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:23:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:23:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:23:39.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:23:39.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24b2b549-c284-462b-a87c-47ae2c777fb2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:39.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24b2b549-c284-462b-a87c-47ae2c777fb2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b5ee39b 2021-05-04T01:23:39.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24b2b549-c284-462b-a87c-47ae2c777fb2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:39.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24b2b549-c284-462b-a87c-47ae2c777fb2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:23:39.853Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=24b2b549-c284-462b-a87c-47ae2c777fb2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:23:39.854Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=24b2b549-c284-462b-a87c-47ae2c777fb2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:23:39.854Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=24b2b549-c284-462b-a87c-47ae2c777fb2 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:23:39.854Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=24b2b549-c284-462b-a87c-47ae2c777fb2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:23:40.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c5543de-2034-481e-ac33-f211c9e01e19 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:40.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c5543de-2034-481e-ac33-f211c9e01e19 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73cc2a18 2021-05-04T01:23:40.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c5543de-2034-481e-ac33-f211c9e01e19 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:40.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c5543de-2034-481e-ac33-f211c9e01e19 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:23:40.765Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7c5543de-2034-481e-ac33-f211c9e01e19 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:23:40.766Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7c5543de-2034-481e-ac33-f211c9e01e19 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:23:41.211Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:23:42.277Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70d48239-610d-4479-bbd2-77d27701bbdf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:42.277Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70d48239-610d-4479-bbd2-77d27701bbdf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54e0122a 2021-05-04T01:23:42.277Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70d48239-610d-4479-bbd2-77d27701bbdf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:42.277Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70d48239-610d-4479-bbd2-77d27701bbdf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:23:42.280Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=70d48239-610d-4479-bbd2-77d27701bbdf ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:23:42.280Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=70d48239-610d-4479-bbd2-77d27701bbdf ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:23:43.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9903684d-da13-4536-91ad-2d15ac8599a8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:43.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9903684d-da13-4536-91ad-2d15ac8599a8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76c50cb9 2021-05-04T01:23:43.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9903684d-da13-4536-91ad-2d15ac8599a8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:43.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9903684d-da13-4536-91ad-2d15ac8599a8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:23:43.205Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=9903684d-da13-4536-91ad-2d15ac8599a8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:23:43.205Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=9903684d-da13-4536-91ad-2d15ac8599a8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:23:43.206Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9903684d-da13-4536-91ad-2d15ac8599a8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:23:43.206Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9903684d-da13-4536-91ad-2d15ac8599a8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:23:45.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff2b0961-8779-43ce-89df-4565277af74d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:45.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff2b0961-8779-43ce-89df-4565277af74d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49a1c8fe 2021-05-04T01:23:45.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff2b0961-8779-43ce-89df-4565277af74d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:45.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff2b0961-8779-43ce-89df-4565277af74d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:23:48.887Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=ff2b0961-8779-43ce-89df-4565277af74d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:23:48.888Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=ff2b0961-8779-43ce-89df-4565277af74d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:23:49.623Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d439ed3e-69bc-47d6-964a-0013ed2f6e45 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:49.624Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d439ed3e-69bc-47d6-964a-0013ed2f6e45 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@289d30df 2021-05-04T01:23:49.624Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d439ed3e-69bc-47d6-964a-0013ed2f6e45 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:49.624Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d439ed3e-69bc-47d6-964a-0013ed2f6e45 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:23:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:23:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:50.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:23:50.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:23:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:23:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:23:50.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:23:52.633Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d439ed3e-69bc-47d6-964a-0013ed2f6e45 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:23:52.634Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d439ed3e-69bc-47d6-964a-0013ed2f6e45 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:23:54.289Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5abc9f54-962b-41b9-92f2-49fd47a948bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:54.289Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5abc9f54-962b-41b9-92f2-49fd47a948bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55c3e428 2021-05-04T01:23:54.289Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5abc9f54-962b-41b9-92f2-49fd47a948bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:54.289Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5abc9f54-962b-41b9-92f2-49fd47a948bb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:23:57.301Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5abc9f54-962b-41b9-92f2-49fd47a948bb ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:23:57.302Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5abc9f54-962b-41b9-92f2-49fd47a948bb ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:23:59.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56bc2806-c817-4947-b1bb-7976769a2d11 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:23:59.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56bc2806-c817-4947-b1bb-7976769a2d11 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78c5a0e0 2021-05-04T01:23:59.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56bc2806-c817-4947-b1bb-7976769a2d11 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:23:59.953Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56bc2806-c817-4947-b1bb-7976769a2d11 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:02.971Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=56bc2806-c817-4947-b1bb-7976769a2d11 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:24:02.971Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=56bc2806-c817-4947-b1bb-7976769a2d11 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:24:03.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6ce23bb-d911-4663-ab69-cf187804284e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:03.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6ce23bb-d911-4663-ab69-cf187804284e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3149e982 2021-05-04T01:24:03.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6ce23bb-d911-4663-ab69-cf187804284e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:03.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6ce23bb-d911-4663-ab69-cf187804284e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:24:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:05.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:24:05.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:24:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:05.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:24:06.714Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d6ce23bb-d911-4663-ab69-cf187804284e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:24:06.715Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d6ce23bb-d911-4663-ab69-cf187804284e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:24:07.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba92cbea-2dcd-4a14-8021-bc0b88331832 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:07.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba92cbea-2dcd-4a14-8021-bc0b88331832 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26a57e9d 2021-05-04T01:24:07.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba92cbea-2dcd-4a14-8021-bc0b88331832 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:07.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba92cbea-2dcd-4a14-8021-bc0b88331832 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:10.411Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=ba92cbea-2dcd-4a14-8021-bc0b88331832 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:24:10.411Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=ba92cbea-2dcd-4a14-8021-bc0b88331832 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:24:10.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c7ae79-466a-4be9-8c57-491a969e5c5a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:10.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c7ae79-466a-4be9-8c57-491a969e5c5a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ecadee6 2021-05-04T01:24:10.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c7ae79-466a-4be9-8c57-491a969e5c5a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:10.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c7ae79-466a-4be9-8c57-491a969e5c5a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:11.939Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f0c7ae79-466a-4be9-8c57-491a969e5c5a ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:11.939Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f0c7ae79-466a-4be9-8c57-491a969e5c5a ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:12.392Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14fc30dd-0425-4efc-ab57-b2a66b670d24 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:12.392Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14fc30dd-0425-4efc-ab57-b2a66b670d24 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40d268f4 2021-05-04T01:24:12.392Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14fc30dd-0425-4efc-ab57-b2a66b670d24 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:12.392Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14fc30dd-0425-4efc-ab57-b2a66b670d24 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:14.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91d33267-bc0d-49c0-87a1-27ac8b588729 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:14.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91d33267-bc0d-49c0-87a1-27ac8b588729 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19fe7505 2021-05-04T01:24:14.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91d33267-bc0d-49c0-87a1-27ac8b588729 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:14.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91d33267-bc0d-49c0-87a1-27ac8b588729 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:15.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26de6911-f0db-49d6-9448-95b1d5602fd6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:15.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26de6911-f0db-49d6-9448-95b1d5602fd6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@613ba532 2021-05-04T01:24:15.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26de6911-f0db-49d6-9448-95b1d5602fd6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:15.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26de6911-f0db-49d6-9448-95b1d5602fd6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:17.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6910fe7-ce38-4914-bc67-5ff8cd9a9a93 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:17.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6910fe7-ce38-4914-bc67-5ff8cd9a9a93 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59bfd38d 2021-05-04T01:24:17.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6910fe7-ce38-4914-bc67-5ff8cd9a9a93 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:17.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6910fe7-ce38-4914-bc67-5ff8cd9a9a93 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:24:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:20.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:24:20.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:24:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:20.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:24:20.971Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e6910fe7-ce38-4914-bc67-5ff8cd9a9a93 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:24:20.971Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e6910fe7-ce38-4914-bc67-5ff8cd9a9a93 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:24:20.972Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e6910fe7-ce38-4914-bc67-5ff8cd9a9a93 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:20.972Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e6910fe7-ce38-4914-bc67-5ff8cd9a9a93 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:22.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22583ea2-339f-4e0a-aad0-5c453724327d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:22.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22583ea2-339f-4e0a-aad0-5c453724327d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@af99586 2021-05-04T01:24:22.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22583ea2-339f-4e0a-aad0-5c453724327d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:22.051Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22583ea2-339f-4e0a-aad0-5c453724327d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:22.507Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=22583ea2-339f-4e0a-aad0-5c453724327d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:22.508Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=22583ea2-339f-4e0a-aad0-5c453724327d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:23.548Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=159c1904-5d40-44bd-ae4b-d03d1dcc3116 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:23.548Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=159c1904-5d40-44bd-ae4b-d03d1dcc3116 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c6ba48e 2021-05-04T01:24:23.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=159c1904-5d40-44bd-ae4b-d03d1dcc3116 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:23.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=159c1904-5d40-44bd-ae4b-d03d1dcc3116 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:24:26.554Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=159c1904-5d40-44bd-ae4b-d03d1dcc3116 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:24:26.554Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=159c1904-5d40-44bd-ae4b-d03d1dcc3116 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:24:26.554Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=159c1904-5d40-44bd-ae4b-d03d1dcc3116 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:26.554Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=159c1904-5d40-44bd-ae4b-d03d1dcc3116 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:24:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:31.562Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:31.564Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:24:35.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:35.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:35.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:24:35.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:24:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:35.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:24:36.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:24:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:24:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:24:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:24:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:37.117Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:24:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:24:37.118Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:24:37.598Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:24:37.599Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:42.203Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:44.604Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:24:44.605Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:45.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:46.626Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:46.630Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:46.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6ca8987-8079-4489-b41e-6b3842c2e319 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:46.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6ca8987-8079-4489-b41e-6b3842c2e319 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b6f2735 2021-05-04T01:24:46.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6ca8987-8079-4489-b41e-6b3842c2e319 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:46.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6ca8987-8079-4489-b41e-6b3842c2e319 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:24:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:48.612Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:24:48.612Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:49.681Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=b6ca8987-8079-4489-b41e-6b3842c2e319 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:24:49.681Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=b6ca8987-8079-4489-b41e-6b3842c2e319 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:24:49.682Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b6ca8987-8079-4489-b41e-6b3842c2e319 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:49.682Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=b6ca8987-8079-4489-b41e-6b3842c2e319 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:24:49.685Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:49.687Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:49.687Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8574d98-ec71-4c68-bb0b-455bb217dbdd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:49.687Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8574d98-ec71-4c68-bb0b-455bb217dbdd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7189e1d2 2021-05-04T01:24:49.687Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8574d98-ec71-4c68-bb0b-455bb217dbdd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:49.687Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a8574d98-ec71-4c68-bb0b-455bb217dbdd PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:24:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:24:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:50.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:24:50.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:24:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:24:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:24:50.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:24:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:52.743Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a8574d98-ec71-4c68-bb0b-455bb217dbdd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:24:52.744Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a8574d98-ec71-4c68-bb0b-455bb217dbdd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:24:52.744Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a8574d98-ec71-4c68-bb0b-455bb217dbdd ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:24:52.744Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=a8574d98-ec71-4c68-bb0b-455bb217dbdd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:24:52.800Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:52.803Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:52.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=280e24ac-f4f5-4205-8c59-1383d46e6816 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:52.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=280e24ac-f4f5-4205-8c59-1383d46e6816 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7293b30b 2021-05-04T01:24:52.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=280e24ac-f4f5-4205-8c59-1383d46e6816 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:52.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=280e24ac-f4f5-4205-8c59-1383d46e6816 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:24:54.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:55.839Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=280e24ac-f4f5-4205-8c59-1383d46e6816 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:24:55.839Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=280e24ac-f4f5-4205-8c59-1383d46e6816 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:24:55.876Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:55.913Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:55.913Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c38f7e5-1552-4fa3-9319-bdb6f84cc1f3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:55.914Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c38f7e5-1552-4fa3-9319-bdb6f84cc1f3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17cef1af 2021-05-04T01:24:55.914Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c38f7e5-1552-4fa3-9319-bdb6f84cc1f3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:55.914Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c38f7e5-1552-4fa3-9319-bdb6f84cc1f3 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:24:56.634Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:24:56.635Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 16000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:24:58.979Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7c38f7e5-1552-4fa3-9319-bdb6f84cc1f3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:24:58.979Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7c38f7e5-1552-4fa3-9319-bdb6f84cc1f3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:24:58.983Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:58.986Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:24:58.986Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97c2b0ab-33b6-4468-b82d-1b0211e2dcb0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:24:58.986Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97c2b0ab-33b6-4468-b82d-1b0211e2dcb0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3383d63f 2021-05-04T01:24:58.986Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97c2b0ab-33b6-4468-b82d-1b0211e2dcb0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:24:58.986Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97c2b0ab-33b6-4468-b82d-1b0211e2dcb0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:25:02.042Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=97c2b0ab-33b6-4468-b82d-1b0211e2dcb0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:25:02.042Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=97c2b0ab-33b6-4468-b82d-1b0211e2dcb0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:25:02.101Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:02.103Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:02.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fbdf779-98e6-446f-b3ed-92a3bb7c8674 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:02.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fbdf779-98e6-446f-b3ed-92a3bb7c8674 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42577a97 2021-05-04T01:25:02.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fbdf779-98e6-446f-b3ed-92a3bb7c8674 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:02.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fbdf779-98e6-446f-b3ed-92a3bb7c8674 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:03.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:25:03.979Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.000Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6afeeb76-f895-49ed-8bd6-eac4cc16397d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:04.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6afeeb76-f895-49ed-8bd6-eac4cc16397d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@aa0c25d 2021-05-04T01:25:04.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6afeeb76-f895-49ed-8bd6-eac4cc16397d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:04.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6afeeb76-f895-49ed-8bd6-eac4cc16397d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:04.027Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.029Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.029Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbf750af-09aa-4dfd-a4b0-2180f53a0f1f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:04.029Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbf750af-09aa-4dfd-a4b0-2180f53a0f1f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d744c2a 2021-05-04T01:25:04.029Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbf750af-09aa-4dfd-a4b0-2180f53a0f1f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:04.029Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbf750af-09aa-4dfd-a4b0-2180f53a0f1f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:04.132Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.134Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.134Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee3357d3-6a65-455b-821d-6252a815b0a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:04.134Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee3357d3-6a65-455b-821d-6252a815b0a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@748e0cd1 2021-05-04T01:25:04.134Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee3357d3-6a65-455b-821d-6252a815b0a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:04.134Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee3357d3-6a65-455b-821d-6252a815b0a0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:04.220Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.223Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97f7e772-b84e-4984-9a70-caba548b7ee5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:04.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97f7e772-b84e-4984-9a70-caba548b7ee5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2dbd6a69 2021-05-04T01:25:04.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97f7e772-b84e-4984-9a70-caba548b7ee5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:04.223Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97f7e772-b84e-4984-9a70-caba548b7ee5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:04.408Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.410Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a668093-ac87-44ac-ab11-6daf9b9db6d1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:04.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a668093-ac87-44ac-ab11-6daf9b9db6d1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2183e7a9 2021-05-04T01:25:04.411Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a668093-ac87-44ac-ab11-6daf9b9db6d1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:04.411Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a668093-ac87-44ac-ab11-6daf9b9db6d1 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:04.520Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.522Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14b62586-56ff-40f4-a905-ab4d4afa20c4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:04.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14b62586-56ff-40f4-a905-ab4d4afa20c4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e39eaaf 2021-05-04T01:25:04.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14b62586-56ff-40f4-a905-ab4d4afa20c4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:04.523Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14b62586-56ff-40f4-a905-ab4d4afa20c4 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:04.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.702Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea160195-6659-4e52-847e-442b0a69d422 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:04.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea160195-6659-4e52-847e-442b0a69d422 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ad5973d 2021-05-04T01:25:04.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea160195-6659-4e52-847e-442b0a69d422 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:04.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea160195-6659-4e52-847e-442b0a69d422 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:04.800Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.802Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf07023f-fb2d-4324-bb39-db6028d72af7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:04.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf07023f-fb2d-4324-bb39-db6028d72af7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c7d4a2c 2021-05-04T01:25:04.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf07023f-fb2d-4324-bb39-db6028d72af7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:04.803Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf07023f-fb2d-4324-bb39-db6028d72af7 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:04.900Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.901Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:04.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f17c16ec-ea80-43ed-84d4-231cc3814208 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:04.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f17c16ec-ea80-43ed-84d4-231cc3814208 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72fd42a5 2021-05-04T01:25:04.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f17c16ec-ea80-43ed-84d4-231cc3814208 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:04.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f17c16ec-ea80-43ed-84d4-231cc3814208 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:05.000Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.001Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d610f43b-dc2e-4769-a12c-d0507f81ce95 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:05.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d610f43b-dc2e-4769-a12c-d0507f81ce95 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33d0a2c7 2021-05-04T01:25:05.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d610f43b-dc2e-4769-a12c-d0507f81ce95 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:05.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d610f43b-dc2e-4769-a12c-d0507f81ce95 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:05.100Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.102Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=273ee594-b973-47a0-b22f-6d0c5f0e36f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:05.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=273ee594-b973-47a0-b22f-6d0c5f0e36f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@159aa7ff 2021-05-04T01:25:05.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=273ee594-b973-47a0-b22f-6d0c5f0e36f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:05.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=273ee594-b973-47a0-b22f-6d0c5f0e36f4 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:25:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:05.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:25:05.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:25:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:05.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:25:05.201Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.203Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9f302b1-3f48-43fe-a88b-787aa0ed7c53 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:05.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9f302b1-3f48-43fe-a88b-787aa0ed7c53 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3390b34a 2021-05-04T01:25:05.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9f302b1-3f48-43fe-a88b-787aa0ed7c53 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:05.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9f302b1-3f48-43fe-a88b-787aa0ed7c53 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:05.303Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.305Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a7564cb-de1e-46c5-be5d-19479692c247 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:05.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a7564cb-de1e-46c5-be5d-19479692c247 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b6027f7 2021-05-04T01:25:05.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a7564cb-de1e-46c5-be5d-19479692c247 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:05.306Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a7564cb-de1e-46c5-be5d-19479692c247 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:05.400Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.403Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2c1883c-3ccb-43a5-9988-b7200a9572d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2c1883c-3ccb-43a5-9988-b7200a9572d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d906ec6 2021-05-04T01:25:05.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2c1883c-3ccb-43a5-9988-b7200a9572d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:05.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2c1883c-3ccb-43a5-9988-b7200a9572d7 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:05.500Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.502Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cdb3a0e-09fb-4279-b2fa-f0a46ebbe96f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cdb3a0e-09fb-4279-b2fa-f0a46ebbe96f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d7587d4 2021-05-04T01:25:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cdb3a0e-09fb-4279-b2fa-f0a46ebbe96f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:05.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cdb3a0e-09fb-4279-b2fa-f0a46ebbe96f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:05.600Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.603Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f26fe28c-bb8f-4392-b5be-6d1d121156d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:05.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f26fe28c-bb8f-4392-b5be-6d1d121156d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b4975d3 2021-05-04T01:25:05.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f26fe28c-bb8f-4392-b5be-6d1d121156d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:05.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f26fe28c-bb8f-4392-b5be-6d1d121156d7 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:05.701Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.721Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:05.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2cb674-f4d3-47c7-a282-e56846963f42 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:05.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2cb674-f4d3-47c7-a282-e56846963f42 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41bca8d2 2021-05-04T01:25:05.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2cb674-f4d3-47c7-a282-e56846963f42 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:05.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd2cb674-f4d3-47c7-a282-e56846963f42 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:05.806Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:25:07.662Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:07.664Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:07.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2a7d84a-98d8-4911-a5b1-acee99e2dc25 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:07.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2a7d84a-98d8-4911-a5b1-acee99e2dc25 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f44187d 2021-05-04T01:25:07.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2a7d84a-98d8-4911-a5b1-acee99e2dc25 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:07.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2a7d84a-98d8-4911-a5b1-acee99e2dc25 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:25:07.719Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b2a7d84a-98d8-4911-a5b1-acee99e2dc25 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:25:07.719Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=b2a7d84a-98d8-4911-a5b1-acee99e2dc25 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:25:10.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:10.987Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:10.988Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8beeff90-1385-4b34-81b6-8944f53a3bfa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:10.988Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8beeff90-1385-4b34-81b6-8944f53a3bfa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e7e7f66 2021-05-04T01:25:10.988Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8beeff90-1385-4b34-81b6-8944f53a3bfa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:10.988Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8beeff90-1385-4b34-81b6-8944f53a3bfa PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:25:11.025Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8beeff90-1385-4b34-81b6-8944f53a3bfa ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:25:11.025Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8beeff90-1385-4b34-81b6-8944f53a3bfa ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:25:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e158891-239c-4420-98a4-4404e447716d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e158891-239c-4420-98a4-4404e447716d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e397097 2021-05-04T01:25:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e158891-239c-4420-98a4-4404e447716d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:14.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e158891-239c-4420-98a4-4404e447716d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:14.806Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4e158891-239c-4420-98a4-4404e447716d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:25:14.806Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4e158891-239c-4420-98a4-4404e447716d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:25:17.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92191d6d-eba2-4536-a663-7c1be6a1314b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:17.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92191d6d-eba2-4536-a663-7c1be6a1314b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a67a3ba 2021-05-04T01:25:17.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92191d6d-eba2-4536-a663-7c1be6a1314b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:17.423Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92191d6d-eba2-4536-a663-7c1be6a1314b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:25:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:20.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:20.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:20.179Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:25:20.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:25:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:20.186Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:25:20.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b18db53b-85c6-48c2-8e66-0f0a668d3599 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:20.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b18db53b-85c6-48c2-8e66-0f0a668d3599 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@559fd17b 2021-05-04T01:25:20.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b18db53b-85c6-48c2-8e66-0f0a668d3599 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:20.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b18db53b-85c6-48c2-8e66-0f0a668d3599 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:24.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deb49cfc-db6e-4ea3-98a6-cb05e6cfadca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:24.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deb49cfc-db6e-4ea3-98a6-cb05e6cfadca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8c711e1 2021-05-04T01:25:24.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deb49cfc-db6e-4ea3-98a6-cb05e6cfadca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:24.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=deb49cfc-db6e-4ea3-98a6-cb05e6cfadca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:26.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96f05d8d-af8f-483d-9ef6-3ea74b6f9ab9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:26.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96f05d8d-af8f-483d-9ef6-3ea74b6f9ab9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d3f75a6 2021-05-04T01:25:26.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96f05d8d-af8f-483d-9ef6-3ea74b6f9ab9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:26.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96f05d8d-af8f-483d-9ef6-3ea74b6f9ab9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:29.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46556e6e-d290-4b21-9d66-5351d1923478 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:29.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46556e6e-d290-4b21-9d66-5351d1923478 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a5265c8 2021-05-04T01:25:29.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46556e6e-d290-4b21-9d66-5351d1923478 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:29.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46556e6e-d290-4b21-9d66-5351d1923478 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:32.331Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b22a36cb-aa13-4df9-a5cf-cc25062c9d4b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:32.331Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b22a36cb-aa13-4df9-a5cf-cc25062c9d4b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71ed63f0 2021-05-04T01:25:32.331Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b22a36cb-aa13-4df9-a5cf-cc25062c9d4b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:32.331Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b22a36cb-aa13-4df9-a5cf-cc25062c9d4b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:25:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:35.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:25:35.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:25:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:35.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:35.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:35.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:25:35.380Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13720885-5dcc-4bb8-bd95-f85b31e6ecaf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:35.380Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13720885-5dcc-4bb8-bd95-f85b31e6ecaf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41153c35 2021-05-04T01:25:35.380Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13720885-5dcc-4bb8-bd95-f85b31e6ecaf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:35.380Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13720885-5dcc-4bb8-bd95-f85b31e6ecaf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:25:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:25:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:25:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:25:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:25:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:25:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:25:38.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6811604-d69d-4808-b27b-a88fe4357a40 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:38.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6811604-d69d-4808-b27b-a88fe4357a40 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77b1f3f4 2021-05-04T01:25:38.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6811604-d69d-4808-b27b-a88fe4357a40 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:38.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6811604-d69d-4808-b27b-a88fe4357a40 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:41.697Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf08d120-5dbe-4c30-a91c-df0f349a703f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:25:41.698Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf08d120-5dbe-4c30-a91c-df0f349a703f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e84b4ef 2021-05-04T01:25:41.698Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf08d120-5dbe-4c30-a91c-df0f349a703f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:25:41.698Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf08d120-5dbe-4c30-a91c-df0f349a703f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:25:44.704Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bf08d120-5dbe-4c30-a91c-df0f349a703f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:25:44.705Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bf08d120-5dbe-4c30-a91c-df0f349a703f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:25:44.706Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bf08d120-5dbe-4c30-a91c-df0f349a703f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:25:44.706Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bf08d120-5dbe-4c30-a91c-df0f349a703f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:25:45.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:25:45.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:25:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:25:49.714Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:49.714Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:25:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:25:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:50.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:25:50.184Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:25:50.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:50.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:50.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:50.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:50.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:25:50.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:25:50.191Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:25:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:25:54.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:25:55.720Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:25:55.723Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:25:57.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:02.730Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:26:02.732Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:03.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:04.727Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:04.727Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:04.800Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:04.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7b25df-ae97-4253-a9cb-acee814322bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:04.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7b25df-ae97-4253-a9cb-acee814322bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@246cb22c 2021-05-04T01:26:04.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7b25df-ae97-4253-a9cb-acee814322bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:04.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da7b25df-ae97-4253-a9cb-acee814322bb PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:26:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:05.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:26:05.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:26:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:05.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:26:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:06.737Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:26:06.738Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:07.855Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=da7b25df-ae97-4253-a9cb-acee814322bb ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:26:07.856Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=da7b25df-ae97-4253-a9cb-acee814322bb ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:26:07.856Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=da7b25df-ae97-4253-a9cb-acee814322bb ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:26:07.856Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=da7b25df-ae97-4253-a9cb-acee814322bb ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:26:07.859Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:07.861Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:07.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38bfa40e-0e24-485e-a23d-84aa294361b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:07.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38bfa40e-0e24-485e-a23d-84aa294361b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34a6a239 2021-05-04T01:26:07.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38bfa40e-0e24-485e-a23d-84aa294361b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:07.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38bfa40e-0e24-485e-a23d-84aa294361b4 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:10.895Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=38bfa40e-0e24-485e-a23d-84aa294361b4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:26:10.895Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=38bfa40e-0e24-485e-a23d-84aa294361b4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:26:10.895Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=38bfa40e-0e24-485e-a23d-84aa294361b4 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:26:10.896Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=38bfa40e-0e24-485e-a23d-84aa294361b4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:26:10.898Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:10.901Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:10.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df4ded6c-36a1-4f93-9d12-6c5fcd0797fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:10.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df4ded6c-36a1-4f93-9d12-6c5fcd0797fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e227193 2021-05-04T01:26:10.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df4ded6c-36a1-4f93-9d12-6c5fcd0797fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:10.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df4ded6c-36a1-4f93-9d12-6c5fcd0797fa PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:13.935Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=df4ded6c-36a1-4f93-9d12-6c5fcd0797fa ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:26:13.936Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=df4ded6c-36a1-4f93-9d12-6c5fcd0797fa ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:26:13.938Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:14.000Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:14.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=376b1453-9f4f-47ce-a981-5061632b2de0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:14.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=376b1453-9f4f-47ce-a981-5061632b2de0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@259ea357 2021-05-04T01:26:14.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=376b1453-9f4f-47ce-a981-5061632b2de0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:14.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=376b1453-9f4f-47ce-a981-5061632b2de0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:14.743Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:26:14.744Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 16000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:15.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:17.044Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=376b1453-9f4f-47ce-a981-5061632b2de0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:26:17.044Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=376b1453-9f4f-47ce-a981-5061632b2de0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:26:17.047Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:17.049Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:17.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1690a0fa-6b8d-474d-81dd-8328b946e87b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:17.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1690a0fa-6b8d-474d-81dd-8328b946e87b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d2162b 2021-05-04T01:26:17.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1690a0fa-6b8d-474d-81dd-8328b946e87b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:17.050Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1690a0fa-6b8d-474d-81dd-8328b946e87b PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:20.099Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=1690a0fa-6b8d-474d-81dd-8328b946e87b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:26:20.100Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=1690a0fa-6b8d-474d-81dd-8328b946e87b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:26:20.103Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.106Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d797355e-b51f-4b9f-aff2-0c1bf6d5d95a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:20.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d797355e-b51f-4b9f-aff2-0c1bf6d5d95a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@329d8bf7 2021-05-04T01:26:20.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d797355e-b51f-4b9f-aff2-0c1bf6d5d95a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:20.106Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d797355e-b51f-4b9f-aff2-0c1bf6d5d95a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:26:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:20.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:26:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:26:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:20.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:26:20.581Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.584Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.584Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c31b22e-f618-4f39-b7ea-4066e4640f52 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:20.584Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c31b22e-f618-4f39-b7ea-4066e4640f52 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@556a0dd5 2021-05-04T01:26:20.584Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c31b22e-f618-4f39-b7ea-4066e4640f52 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:20.584Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c31b22e-f618-4f39-b7ea-4066e4640f52 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:20.639Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.641Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.641Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a921f9c7-1606-454e-bdf3-3d814cb017a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:20.641Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a921f9c7-1606-454e-bdf3-3d814cb017a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f4e9182 2021-05-04T01:26:20.641Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a921f9c7-1606-454e-bdf3-3d814cb017a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:20.641Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a921f9c7-1606-454e-bdf3-3d814cb017a2 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:20.720Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.722Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c29d0a6-9a7a-4220-8070-5694daefc10e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:20.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c29d0a6-9a7a-4220-8070-5694daefc10e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21527faf 2021-05-04T01:26:20.723Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c29d0a6-9a7a-4220-8070-5694daefc10e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:20.723Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c29d0a6-9a7a-4220-8070-5694daefc10e PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:20.824Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.826Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26c3c45-6c3f-4a37-8918-aef77d83fdac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:20.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26c3c45-6c3f-4a37-8918-aef77d83fdac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a0206 2021-05-04T01:26:20.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26c3c45-6c3f-4a37-8918-aef77d83fdac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:20.827Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a26c3c45-6c3f-4a37-8918-aef77d83fdac PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:20.932Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.935Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:20.935Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1abcca20-680b-4072-9d72-0a5f49a8c8f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:20.935Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1abcca20-680b-4072-9d72-0a5f49a8c8f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52ba373 2021-05-04T01:26:20.935Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1abcca20-680b-4072-9d72-0a5f49a8c8f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:20.935Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1abcca20-680b-4072-9d72-0a5f49a8c8f9 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:21.023Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:21.025Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:21.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7203de06-8d46-4874-9237-802831087539 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:21.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7203de06-8d46-4874-9237-802831087539 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e62e0d5 2021-05-04T01:26:21.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7203de06-8d46-4874-9237-802831087539 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:21.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7203de06-8d46-4874-9237-802831087539 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:21.145Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:21.147Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:21.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4b41e25-13fb-4ec1-94da-13f38c719f00 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:21.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4b41e25-13fb-4ec1-94da-13f38c719f00 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c7f59b1 2021-05-04T01:26:21.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4b41e25-13fb-4ec1-94da-13f38c719f00 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:21.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4b41e25-13fb-4ec1-94da-13f38c719f00 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:22.706Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:22.709Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:22.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d2dace-e1e2-4518-bd7f-f0525cd05d91 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:22.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d2dace-e1e2-4518-bd7f-f0525cd05d91 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c15560a 2021-05-04T01:26:22.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d2dace-e1e2-4518-bd7f-f0525cd05d91 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:22.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d2dace-e1e2-4518-bd7f-f0525cd05d91 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:24.122Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:26:25.700Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:25.704Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:25.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40521de9-dc3c-4943-b64d-b441a1fba975 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:25.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40521de9-dc3c-4943-b64d-b441a1fba975 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@347d50c0 2021-05-04T01:26:25.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40521de9-dc3c-4943-b64d-b441a1fba975 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:25.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40521de9-dc3c-4943-b64d-b441a1fba975 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:25.808Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=40521de9-dc3c-4943-b64d-b441a1fba975 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:26:25.808Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=40521de9-dc3c-4943-b64d-b441a1fba975 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:26:28.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:28.665Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:28.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aeb2d4a-4c5a-4d78-a0b3-8d8ead33f022 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:28.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aeb2d4a-4c5a-4d78-a0b3-8d8ead33f022 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d807ad2 2021-05-04T01:26:28.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aeb2d4a-4c5a-4d78-a0b3-8d8ead33f022 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:28.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aeb2d4a-4c5a-4d78-a0b3-8d8ead33f022 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:30.799Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:26:30.801Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 32000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:31.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:31.665Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:31.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8369a285-65b8-413c-9d07-f78c9010e576 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:31.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8369a285-65b8-413c-9d07-f78c9010e576 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6bf40c02 2021-05-04T01:26:31.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8369a285-65b8-413c-9d07-f78c9010e576 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:31.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8369a285-65b8-413c-9d07-f78c9010e576 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:34.701Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:34.703Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:34.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6126af6a-1e95-4a4e-8d02-7a32b19086b5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:34.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6126af6a-1e95-4a4e-8d02-7a32b19086b5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d97a270 2021-05-04T01:26:34.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6126af6a-1e95-4a4e-8d02-7a32b19086b5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:34.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6126af6a-1e95-4a4e-8d02-7a32b19086b5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:26:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:26:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:35.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:26:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:26:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:35.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:26:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:26:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:26:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:26:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:26:37.109Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:37.109Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:37.113Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:26:37.114Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:26:37.114Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:26:37.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:38.476Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:38.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1cbdcdc-d047-4c65-8f62-f4d8831d0588 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:38.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1cbdcdc-d047-4c65-8f62-f4d8831d0588 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73b21c52 2021-05-04T01:26:38.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1cbdcdc-d047-4c65-8f62-f4d8831d0588 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:38.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1cbdcdc-d047-4c65-8f62-f4d8831d0588 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:26:38.527Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b1cbdcdc-d047-4c65-8f62-f4d8831d0588 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:26:38.527Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b1cbdcdc-d047-4c65-8f62-f4d8831d0588 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:26:40.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:43.200Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:43.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69673e5f-31b7-49a7-95f0-b7200c803b23 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:43.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69673e5f-31b7-49a7-95f0-b7200c803b23 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d4aae1c 2021-05-04T01:26:43.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69673e5f-31b7-49a7-95f0-b7200c803b23 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:43.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69673e5f-31b7-49a7-95f0-b7200c803b23 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:26:43.265Z [JanusGraph-Health-Check-Task] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$VertexConstructor.get(StandardJanusGraphTx.java:462) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$VertexConstructor.get(StandardJanusGraphTx.java:440) at org.janusgraph.graphdb.transaction.vertexcache.GuavaVertexCache.get(GuavaVertexCache.java:77) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getVertex(StandardJanusGraphTx.java:387) at org.janusgraph.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:53) at org.janusgraph.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:37) at org.janusgraph.graphdb.internal.AbstractElement.isRemoved(AbstractElement.java:141) at org.janusgraph.graphdb.vertices.AbstractVertex.verifyAccess(AbstractVertex.java:89) at org.janusgraph.graphdb.vertices.AbstractVertex.query(AbstractVertex.java:137) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.isGraphOpen(JanusGraphClient.java:346) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckScheduledTask.run(JanusGraphClient.java:95) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:26:43.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:43.867Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:826) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:26:44.101Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:44.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ba837dc-68a2-4afa-ac79-054ad097202a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:44.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ba837dc-68a2-4afa-ac79-054ad097202a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16153dab 2021-05-04T01:26:44.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ba837dc-68a2-4afa-ac79-054ad097202a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:44.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ba837dc-68a2-4afa-ac79-054ad097202a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:26:46.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:47.091Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:47.091Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7fe8e6-0db2-420e-8a3e-55750b08868d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:26:47.091Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7fe8e6-0db2-420e-8a3e-55750b08868d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@681993b 2021-05-04T01:26:47.091Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7fe8e6-0db2-420e-8a3e-55750b08868d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:26:47.092Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7fe8e6-0db2-420e-8a3e-55750b08868d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:26:49.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:26:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:26:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:50.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:26:50.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:26:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:26:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:26:50.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:26:56.401Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:56.402Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:26:57.402Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:57.659Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:26:40Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:26:58.402Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:26:59.403Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:00.403Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:01.403Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:01.932Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:01.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12584d73-b5a9-4b37-99ca-ac3910f8a06a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:01.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12584d73-b5a9-4b37-99ca-ac3910f8a06a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65324535 2021-05-04T01:27:01.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12584d73-b5a9-4b37-99ca-ac3910f8a06a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:01.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12584d73-b5a9-4b37-99ca-ac3910f8a06a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:27:02.406Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:02.809Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:27:02.810Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 64000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:02.926Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:27:02.926Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:832) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:27:03.406Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:27:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:05.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:27:05.186Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:27:05.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:05.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:05.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:05.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:05.191Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:05.191Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:05.193Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:27:05.301Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:27:06.482Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbc2d05a-af5f-457a-bfaa-7d73429e59a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:06.482Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbc2d05a-af5f-457a-bfaa-7d73429e59a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d802dc1 2021-05-04T01:27:06.482Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbc2d05a-af5f-457a-bfaa-7d73429e59a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:06.482Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbc2d05a-af5f-457a-bfaa-7d73429e59a6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:27:09.489Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bbc2d05a-af5f-457a-bfaa-7d73429e59a6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:27:09.489Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bbc2d05a-af5f-457a-bfaa-7d73429e59a6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:27:09.490Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bbc2d05a-af5f-457a-bfaa-7d73429e59a6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:09.490Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=bbc2d05a-af5f-457a-bfaa-7d73429e59a6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:27:09.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8952c16c-3d4b-457e-a2db-c24670f9e156 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:09.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8952c16c-3d4b-457e-a2db-c24670f9e156 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52c7bdbf 2021-05-04T01:27:09.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8952c16c-3d4b-457e-a2db-c24670f9e156 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:09.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8952c16c-3d4b-457e-a2db-c24670f9e156 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:27:10.109Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8952c16c-3d4b-457e-a2db-c24670f9e156 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:10.109Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8952c16c-3d4b-457e-a2db-c24670f9e156 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:10.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8134834c-b91f-4f9c-8f68-72a06ba1a38d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:10.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8134834c-b91f-4f9c-8f68-72a06ba1a38d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2fe94355 2021-05-04T01:27:10.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8134834c-b91f-4f9c-8f68-72a06ba1a38d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:10.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8134834c-b91f-4f9c-8f68-72a06ba1a38d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:27:13.375Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8134834c-b91f-4f9c-8f68-72a06ba1a38d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:27:13.375Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8134834c-b91f-4f9c-8f68-72a06ba1a38d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:27:13.375Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8134834c-b91f-4f9c-8f68-72a06ba1a38d ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:13.376Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8134834c-b91f-4f9c-8f68-72a06ba1a38d ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:14.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bdbcc6b-197e-46dd-8e25-8d088f15f85f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:14.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bdbcc6b-197e-46dd-8e25-8d088f15f85f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@436b9d4c 2021-05-04T01:27:14.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bdbcc6b-197e-46dd-8e25-8d088f15f85f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:14.206Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bdbcc6b-197e-46dd-8e25-8d088f15f85f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:27:17.241Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8bdbcc6b-197e-46dd-8e25-8d088f15f85f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:27:17.241Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8bdbcc6b-197e-46dd-8e25-8d088f15f85f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:27:19.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=493865a1-0f53-489a-855e-a9c731b7b0f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:19.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=493865a1-0f53-489a-855e-a9c731b7b0f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59aa5895 2021-05-04T01:27:19.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=493865a1-0f53-489a-855e-a9c731b7b0f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:19.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=493865a1-0f53-489a-855e-a9c731b7b0f6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:27:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:27:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:20.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:27:20.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:27:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:20.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:27:21.902Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=493865a1-0f53-489a-855e-a9c731b7b0f6 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:21.903Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=493865a1-0f53-489a-855e-a9c731b7b0f6 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:22.628Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:23.319Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:23.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80ba7527-c4fa-44bc-ae2c-2923af94c246 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:23.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80ba7527-c4fa-44bc-ae2c-2923af94c246 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23a22038 2021-05-04T01:27:23.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80ba7527-c4fa-44bc-ae2c-2923af94c246 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:23.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80ba7527-c4fa-44bc-ae2c-2923af94c246 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:27:24.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:24.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:27:26.328Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=80ba7527-c4fa-44bc-ae2c-2923af94c246 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:27:26.329Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=80ba7527-c4fa-44bc-ae2c-2923af94c246 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:27:26.329Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=80ba7527-c4fa-44bc-ae2c-2923af94c246 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:26.329Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=80ba7527-c4fa-44bc-ae2c-2923af94c246 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:26.333Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:26.872Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:26.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86da80e4-34b0-4d6e-8226-3910d1617dd1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:26.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86da80e4-34b0-4d6e-8226-3910d1617dd1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7801fea5 2021-05-04T01:27:26.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86da80e4-34b0-4d6e-8226-3910d1617dd1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:26.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86da80e4-34b0-4d6e-8226-3910d1617dd1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:27:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:28.348Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:27:29.908Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=86da80e4-34b0-4d6e-8226-3910d1617dd1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:27:29.908Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=86da80e4-34b0-4d6e-8226-3910d1617dd1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:27:29.908Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=86da80e4-34b0-4d6e-8226-3910d1617dd1 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:29.908Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=86da80e4-34b0-4d6e-8226-3910d1617dd1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:27:29.912Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:35.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:27:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:35.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:27:35.182Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:27:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:35.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:35.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:35.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:35.191Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:27:35.191Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:35.193Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:27:36.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:27:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:27:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:27:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:27:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:27:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:27:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:27:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:27:39.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:45.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:49.927Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:49.932Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:49.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b33c24f1-9c58-40ba-829f-2c2b5ab76d6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:49.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b33c24f1-9c58-40ba-829f-2c2b5ab76d6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d5ebe09 2021-05-04T01:27:49.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b33c24f1-9c58-40ba-829f-2c2b5ab76d6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:49.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b33c24f1-9c58-40ba-829f-2c2b5ab76d6a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:27:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:27:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:50.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:27:50.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:27:50.199Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:50.199Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:50.200Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:50.201Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:50.202Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:27:50.202Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:27:50.204Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:27:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:52.967Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=b33c24f1-9c58-40ba-829f-2c2b5ab76d6a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:27:52.967Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=b33c24f1-9c58-40ba-829f-2c2b5ab76d6a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:27:52.967Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b33c24f1-9c58-40ba-829f-2c2b5ab76d6a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:27:52.968Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=b33c24f1-9c58-40ba-829f-2c2b5ab76d6a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:27:53.000Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:53.003Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:53.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ed142b3-9aa2-41e3-b706-b0fd7fea23dd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:27:53.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ed142b3-9aa2-41e3-b706-b0fd7fea23dd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@684cc9e0 2021-05-04T01:27:53.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ed142b3-9aa2-41e3-b706-b0fd7fea23dd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:27:53.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ed142b3-9aa2-41e3-b706-b0fd7fea23dd PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:27:54.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:27:56.033Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6ed142b3-9aa2-41e3-b706-b0fd7fea23dd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:27:56.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6ed142b3-9aa2-41e3-b706-b0fd7fea23dd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:27:56.101Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:27:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:03.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:28:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:05.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:28:05.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:28:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:05.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:05.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:05.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:05.186Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:28:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:06.817Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:28:06.818Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 128000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:08.061Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:08.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69a8cafa-b1b0-4977-b383-0fc41cac5c62 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:08.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69a8cafa-b1b0-4977-b383-0fc41cac5c62 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60f0fea9 2021-05-04T01:28:08.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69a8cafa-b1b0-4977-b383-0fc41cac5c62 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:08.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69a8cafa-b1b0-4977-b383-0fc41cac5c62 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:08.181Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=69a8cafa-b1b0-4977-b383-0fc41cac5c62 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:28:08.181Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=69a8cafa-b1b0-4977-b383-0fc41cac5c62 ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:28:08.183Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:08.503Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:28:09.159Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:09.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0f38670-6200-40de-99f7-a41c51ac01ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:09.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0f38670-6200-40de-99f7-a41c51ac01ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e110ccf 2021-05-04T01:28:09.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0f38670-6200-40de-99f7-a41c51ac01ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:09.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0f38670-6200-40de-99f7-a41c51ac01ca PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:09.406Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:10.656Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:10.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd65cbf0-624f-4a86-820f-b255113e64e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:10.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd65cbf0-624f-4a86-820f-b255113e64e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@310805a 2021-05-04T01:28:10.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd65cbf0-624f-4a86-820f-b255113e64e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:10.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd65cbf0-624f-4a86-820f-b255113e64e9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:10.868Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cd65cbf0-624f-4a86-820f-b255113e64e9 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:28:10.868Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cd65cbf0-624f-4a86-820f-b255113e64e9 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:28:10.871Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:12.657Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:12.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3522158a-b0ea-41b4-9058-d18091eb63c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:12.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3522158a-b0ea-41b4-9058-d18091eb63c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64722327 2021-05-04T01:28:12.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3522158a-b0ea-41b4-9058-d18091eb63c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:12.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3522158a-b0ea-41b4-9058-d18091eb63c9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:13.108Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:15.800Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:15.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=430131da-b673-4344-971f-d12f54757d03 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:15.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=430131da-b673-4344-971f-d12f54757d03 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23f6a22 2021-05-04T01:28:15.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=430131da-b673-4344-971f-d12f54757d03 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:15.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=430131da-b673-4344-971f-d12f54757d03 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:15.934Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:18.565Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:18.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8fe1454-bd89-4c24-9d39-f8ec14a18b31 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:18.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8fe1454-bd89-4c24-9d39-f8ec14a18b31 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ce79de 2021-05-04T01:28:18.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8fe1454-bd89-4c24-9d39-f8ec14a18b31 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:18.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8fe1454-bd89-4c24-9d39-f8ec14a18b31 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:28:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:20.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:28:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:28:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:20.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:28:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:28:21.567Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f8fe1454-bd89-4c24-9d39-f8ec14a18b31 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:28:21.567Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f8fe1454-bd89-4c24-9d39-f8ec14a18b31 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:28:21.600Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:24.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:33.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:35.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:28:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:35.179Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:28:35.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:28:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:35.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:35.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:35.188Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:28:36.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:28:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:28:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:28:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:28:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:28:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:28:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:28:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:40.255Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:40.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5f2d230-6f01-463b-afe7-7efb6a6c3884 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:40.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5f2d230-6f01-463b-afe7-7efb6a6c3884 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3db3b4d 2021-05-04T01:28:40.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5f2d230-6f01-463b-afe7-7efb6a6c3884 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:40.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5f2d230-6f01-463b-afe7-7efb6a6c3884 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:40.262Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a5f2d230-6f01-463b-afe7-7efb6a6c3884 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:28:40.263Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a5f2d230-6f01-463b-afe7-7efb6a6c3884 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:28:40.263Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a5f2d230-6f01-463b-afe7-7efb6a6c3884 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:28:40.264Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=a5f2d230-6f01-463b-afe7-7efb6a6c3884 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:28:40.266Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:41.101Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:41.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1444964c-aaaa-4221-96d0-99ae0abe6130 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:41.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1444964c-aaaa-4221-96d0-99ae0abe6130 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@fd2518a 2021-05-04T01:28:41.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1444964c-aaaa-4221-96d0-99ae0abe6130 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:41.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1444964c-aaaa-4221-96d0-99ae0abe6130 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:41.104Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=1444964c-aaaa-4221-96d0-99ae0abe6130 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:28:41.105Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=1444964c-aaaa-4221-96d0-99ae0abe6130 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:28:41.108Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:41.300Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:28:42.601Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:42.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=849301d2-ee48-49cd-a99b-a14a6778375a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:42.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=849301d2-ee48-49cd-a99b-a14a6778375a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@227e842d 2021-05-04T01:28:42.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=849301d2-ee48-49cd-a99b-a14a6778375a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:42.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=849301d2-ee48-49cd-a99b-a14a6778375a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:42.604Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=849301d2-ee48-49cd-a99b-a14a6778375a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:28:42.604Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=849301d2-ee48-49cd-a99b-a14a6778375a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:28:42.608Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:44.400Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:44.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1148ed5-dc4b-4735-8656-39b168e4667b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:28:44.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1148ed5-dc4b-4735-8656-39b168e4667b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@637c562f 2021-05-04T01:28:44.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1148ed5-dc4b-4735-8656-39b168e4667b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:28:44.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1148ed5-dc4b-4735-8656-39b168e4667b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:28:47.411Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c1148ed5-dc4b-4735-8656-39b168e4667b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:28:47.412Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c1148ed5-dc4b-4735-8656-39b168e4667b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:28:47.412Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c1148ed5-dc4b-4735-8656-39b168e4667b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:28:47.412Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c1148ed5-dc4b-4735-8656-39b168e4667b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:28:47.415Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:28:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:28:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:28:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:50.179Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:28:50.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:28:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:28:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:28:50.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:28:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:28:54.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:28:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:02.611Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:29:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:29:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:05.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:29:05.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:29:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:29:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:05.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:29:05.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:05.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:29:08.238Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.243Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.243Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fb5f909-3a09-45b6-988b-2b26baf34383 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.243Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fb5f909-3a09-45b6-988b-2b26baf34383 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@634f62ef 2021-05-04T01:29:08.243Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fb5f909-3a09-45b6-988b-2b26baf34383 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.243Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fb5f909-3a09-45b6-988b-2b26baf34383 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.371Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5fb5f909-3a09-45b6-988b-2b26baf34383 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:29:08.371Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=5fb5f909-3a09-45b6-988b-2b26baf34383 ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:29:08.375Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.377Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.378Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a515ca16-f9da-4874-8a68-6c299654200f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.378Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a515ca16-f9da-4874-8a68-6c299654200f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@640903d2 2021-05-04T01:29:08.378Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a515ca16-f9da-4874-8a68-6c299654200f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.378Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a515ca16-f9da-4874-8a68-6c299654200f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.385Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.388Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.388Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a53eb484-87c2-4858-8dc2-c9af9c7ce1fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.388Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a53eb484-87c2-4858-8dc2-c9af9c7ce1fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67f00232 2021-05-04T01:29:08.388Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a53eb484-87c2-4858-8dc2-c9af9c7ce1fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.388Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a53eb484-87c2-4858-8dc2-c9af9c7ce1fe PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.393Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.395Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.395Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f4958f-f9bb-4f81-a3ae-49e3833d8d8b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.395Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f4958f-f9bb-4f81-a3ae-49e3833d8d8b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fb05c39 2021-05-04T01:29:08.395Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f4958f-f9bb-4f81-a3ae-49e3833d8d8b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.395Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f4958f-f9bb-4f81-a3ae-49e3833d8d8b PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.400Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.401Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e77194-07c9-4511-a1e4-f96eadbe7410 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e77194-07c9-4511-a1e4-f96eadbe7410 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35dc5180 2021-05-04T01:29:08.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e77194-07c9-4511-a1e4-f96eadbe7410 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e77194-07c9-4511-a1e4-f96eadbe7410 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.406Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.408Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f40e011-1911-4b67-ad04-1f285d040598 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f40e011-1911-4b67-ad04-1f285d040598 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1967c37a 2021-05-04T01:29:08.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f40e011-1911-4b67-ad04-1f285d040598 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f40e011-1911-4b67-ad04-1f285d040598 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.415Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.417Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3042a8-f2e3-494c-a0a0-49f0c03f15e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3042a8-f2e3-494c-a0a0-49f0c03f15e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ee1b3f6 2021-05-04T01:29:08.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3042a8-f2e3-494c-a0a0-49f0c03f15e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b3042a8-f2e3-494c-a0a0-49f0c03f15e9 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.422Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.424Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.424Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=65e5a0a3-54d7-466a-b924-48804240fb3f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.424Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=65e5a0a3-54d7-466a-b924-48804240fb3f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3cd7987 2021-05-04T01:29:08.424Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=65e5a0a3-54d7-466a-b924-48804240fb3f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.424Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=65e5a0a3-54d7-466a-b924-48804240fb3f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.430Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.432Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.432Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6ddf401-f5ed-4940-a591-b0397e438cd0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.432Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6ddf401-f5ed-4940-a591-b0397e438cd0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54ca782f 2021-05-04T01:29:08.432Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6ddf401-f5ed-4940-a591-b0397e438cd0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.433Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6ddf401-f5ed-4940-a591-b0397e438cd0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.438Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.440Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.440Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a96b8de9-5e3b-4176-9bd7-a0dfbe53ee96 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.440Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a96b8de9-5e3b-4176-9bd7-a0dfbe53ee96 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73911ddf 2021-05-04T01:29:08.440Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a96b8de9-5e3b-4176-9bd7-a0dfbe53ee96 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.440Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a96b8de9-5e3b-4176-9bd7-a0dfbe53ee96 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.445Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.447Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.447Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53d769a8-49ed-4550-9274-b91e14ce4fb6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.447Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53d769a8-49ed-4550-9274-b91e14ce4fb6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9c6e58 2021-05-04T01:29:08.447Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53d769a8-49ed-4550-9274-b91e14ce4fb6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.447Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53d769a8-49ed-4550-9274-b91e14ce4fb6 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.452Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.453Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9208e5d4-8406-465d-a587-4be3a0e30207 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9208e5d4-8406-465d-a587-4be3a0e30207 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68f52e8d 2021-05-04T01:29:08.453Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9208e5d4-8406-465d-a587-4be3a0e30207 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.454Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9208e5d4-8406-465d-a587-4be3a0e30207 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.459Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.460Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c01e045-3f39-49b4-8dd0-f0f47d362d6e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c01e045-3f39-49b4-8dd0-f0f47d362d6e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ff928bc 2021-05-04T01:29:08.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c01e045-3f39-49b4-8dd0-f0f47d362d6e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c01e045-3f39-49b4-8dd0-f0f47d362d6e PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.465Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.466Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=878fe2c9-7297-485e-afab-c433aae36182 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=878fe2c9-7297-485e-afab-c433aae36182 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@147dbf79 2021-05-04T01:29:08.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=878fe2c9-7297-485e-afab-c433aae36182 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=878fe2c9-7297-485e-afab-c433aae36182 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.472Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.473Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.473Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fd6f7de-d4b6-4c2d-95db-7b5db7642bc6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.473Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fd6f7de-d4b6-4c2d-95db-7b5db7642bc6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7770ed00 2021-05-04T01:29:08.473Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fd6f7de-d4b6-4c2d-95db-7b5db7642bc6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.473Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fd6f7de-d4b6-4c2d-95db-7b5db7642bc6 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.477Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.479Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.479Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3032ca7a-19cf-44ce-9857-8c400dd5be5c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.479Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3032ca7a-19cf-44ce-9857-8c400dd5be5c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@36302425 2021-05-04T01:29:08.479Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3032ca7a-19cf-44ce-9857-8c400dd5be5c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.479Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3032ca7a-19cf-44ce-9857-8c400dd5be5c PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.483Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.485Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.485Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0a1def5-dd81-439f-b858-e2be5797bae4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.485Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0a1def5-dd81-439f-b858-e2be5797bae4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@102a4697 2021-05-04T01:29:08.485Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0a1def5-dd81-439f-b858-e2be5797bae4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.485Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0a1def5-dd81-439f-b858-e2be5797bae4 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.490Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.491Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.492Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6159ef6e-c801-4add-9bbd-379017ff273b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.492Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6159ef6e-c801-4add-9bbd-379017ff273b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69d5b59c 2021-05-04T01:29:08.492Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6159ef6e-c801-4add-9bbd-379017ff273b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.492Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6159ef6e-c801-4add-9bbd-379017ff273b PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.496Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.497Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5c6de51-eef2-4284-9f7e-a3f34cb0e534 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5c6de51-eef2-4284-9f7e-a3f34cb0e534 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@193d4d4b 2021-05-04T01:29:08.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5c6de51-eef2-4284-9f7e-a3f34cb0e534 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.498Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5c6de51-eef2-4284-9f7e-a3f34cb0e534 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.502Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.503Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1b5eaadd-68fd-4563-909b-b4f219da540c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1b5eaadd-68fd-4563-909b-b4f219da540c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e1b4345 2021-05-04T01:29:08.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1b5eaadd-68fd-4563-909b-b4f219da540c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1b5eaadd-68fd-4563-909b-b4f219da540c PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.511Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.513Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0063ef43-6a7f-486e-8fe8-56c12ae6876d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0063ef43-6a7f-486e-8fe8-56c12ae6876d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c47b6ef 2021-05-04T01:29:08.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0063ef43-6a7f-486e-8fe8-56c12ae6876d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0063ef43-6a7f-486e-8fe8-56c12ae6876d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.517Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.519Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75134ae8-ae87-4249-8998-14799688e6aa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75134ae8-ae87-4249-8998-14799688e6aa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28fa87ae 2021-05-04T01:29:08.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75134ae8-ae87-4249-8998-14799688e6aa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.519Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75134ae8-ae87-4249-8998-14799688e6aa PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.523Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.525Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a33bca7-c4c2-40a7-b033-0ec807b504c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a33bca7-c4c2-40a7-b033-0ec807b504c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5bdec0ef 2021-05-04T01:29:08.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a33bca7-c4c2-40a7-b033-0ec807b504c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a33bca7-c4c2-40a7-b033-0ec807b504c2 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.530Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.532Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.532Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9e026b-a2df-4d6c-b221-d9ed73dc7f71 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.532Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9e026b-a2df-4d6c-b221-d9ed73dc7f71 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2faba1ba 2021-05-04T01:29:08.532Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9e026b-a2df-4d6c-b221-d9ed73dc7f71 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.532Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9e026b-a2df-4d6c-b221-d9ed73dc7f71 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.536Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.537Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.537Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a653553c-93c2-4421-9f01-c589dc948e80 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.537Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a653553c-93c2-4421-9f01-c589dc948e80 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e8a9a6e 2021-05-04T01:29:08.538Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a653553c-93c2-4421-9f01-c589dc948e80 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.538Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a653553c-93c2-4421-9f01-c589dc948e80 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.542Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.543Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.543Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ebf70cf-de06-4655-9f89-f053f3ee3644 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.543Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ebf70cf-de06-4655-9f89-f053f3ee3644 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b1b5e6 2021-05-04T01:29:08.543Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ebf70cf-de06-4655-9f89-f053f3ee3644 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.543Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ebf70cf-de06-4655-9f89-f053f3ee3644 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:08.548Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.549Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:08.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d503c50d-5126-412b-bde8-6243a1cd8dec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:08.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d503c50d-5126-412b-bde8-6243a1cd8dec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2fc5d4e7 2021-05-04T01:29:08.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d503c50d-5126-412b-bde8-6243a1cd8dec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:08.549Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d503c50d-5126-412b-bde8-6243a1cd8dec PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:10.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:10.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:10.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bddc660d-719c-438b-8745-9dd08d79ae55 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:10.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bddc660d-719c-438b-8745-9dd08d79ae55 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d1116d4 2021-05-04T01:29:10.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bddc660d-719c-438b-8745-9dd08d79ae55 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:10.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bddc660d-719c-438b-8745-9dd08d79ae55 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:13.700Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:13.702Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4211f0c8-42dd-482a-9cce-f40e84bc7ea3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4211f0c8-42dd-482a-9cce-f40e84bc7ea3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77e2bef2 2021-05-04T01:29:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4211f0c8-42dd-482a-9cce-f40e84bc7ea3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4211f0c8-42dd-482a-9cce-f40e84bc7ea3 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:16.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:16.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc01142c-26ac-4ec4-b04e-246b736ef335 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc01142c-26ac-4ec4-b04e-246b736ef335 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78732b6d 2021-05-04T01:29:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc01142c-26ac-4ec4-b04e-246b736ef335 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:16.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc01142c-26ac-4ec4-b04e-246b736ef335 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:19.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:19.666Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=138892f0-a406-4201-be37-718593aec232 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:19.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=138892f0-a406-4201-be37-718593aec232 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7990ba68 2021-05-04T01:29:19.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=138892f0-a406-4201-be37-718593aec232 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:19.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=138892f0-a406-4201-be37-718593aec232 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:29:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:29:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:20.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:20.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:20.179Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:29:20.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:29:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:20.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:20.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:20.188Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:29:22.699Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:22.702Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec603837-e276-49d2-a82c-f23a1b519a35 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:22.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec603837-e276-49d2-a82c-f23a1b519a35 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c62a438 2021-05-04T01:29:22.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec603837-e276-49d2-a82c-f23a1b519a35 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:22.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec603837-e276-49d2-a82c-f23a1b519a35 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:25.816Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:32.901Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:32.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24bbffbe-c82b-4824-90e1-902dc9278960 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:32.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24bbffbe-c82b-4824-90e1-902dc9278960 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e159a68 2021-05-04T01:29:32.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24bbffbe-c82b-4824-90e1-902dc9278960 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:32.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24bbffbe-c82b-4824-90e1-902dc9278960 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:29:32.909Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=24bbffbe-c82b-4824-90e1-902dc9278960 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:29:32.909Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=24bbffbe-c82b-4824-90e1-902dc9278960 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:29:32.912Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:29:34.279Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:34.280Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=605cc322-bf52-4893-94c4-080c4b994fe1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:34.280Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=605cc322-bf52-4893-94c4-080c4b994fe1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f5cf884 2021-05-04T01:29:34.280Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=605cc322-bf52-4893-94c4-080c4b994fe1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:34.280Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=605cc322-bf52-4893-94c4-080c4b994fe1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:29:34.300Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:29:35.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:35.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:35.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:29:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:29:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:35.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:29:36.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:29:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:29:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:29:37.107Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:29:37.109Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:37.109Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:37.113Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:29:37.114Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:29:37.114Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:29:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:45.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:29:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:29:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:50.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:29:50.184Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:29:50.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:50.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:50.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:50.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:50.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:29:50.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:29:50.189Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:29:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:54.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:55.527Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:257) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.528Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:257) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.535Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.536Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae79824a-c81e-4b62-b1dd-e483d480475a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:55.536Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae79824a-c81e-4b62-b1dd-e483d480475a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53e344fd 2021-05-04T01:29:55.536Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae79824a-c81e-4b62-b1dd-e483d480475a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:55.536Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae79824a-c81e-4b62-b1dd-e483d480475a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:55.557Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ae79824a-c81e-4b62-b1dd-e483d480475a ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:29:55.557Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=ae79824a-c81e-4b62-b1dd-e483d480475a ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:29:55.600Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.601Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c637f510-ce40-4c0d-8633-d98234563651 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:55.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c637f510-ce40-4c0d-8633-d98234563651 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a9979dd 2021-05-04T01:29:55.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c637f510-ce40-4c0d-8633-d98234563651 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:55.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c637f510-ce40-4c0d-8633-d98234563651 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:55.606Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.608Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.608Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=339181ac-e119-4f0c-805a-3a8da9be09aa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:55.608Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=339181ac-e119-4f0c-805a-3a8da9be09aa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8bd783c 2021-05-04T01:29:55.608Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=339181ac-e119-4f0c-805a-3a8da9be09aa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:55.608Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=339181ac-e119-4f0c-805a-3a8da9be09aa PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:55.612Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.614Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b88b7f2-6fd8-46d0-9bda-31373e88f3a5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:55.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b88b7f2-6fd8-46d0-9bda-31373e88f3a5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40ed3977 2021-05-04T01:29:55.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b88b7f2-6fd8-46d0-9bda-31373e88f3a5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:55.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b88b7f2-6fd8-46d0-9bda-31373e88f3a5 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:55.618Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.619Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d222ee2a-57ca-4cbe-bec0-77fc915ca663 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:55.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d222ee2a-57ca-4cbe-bec0-77fc915ca663 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9e0493b 2021-05-04T01:29:55.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d222ee2a-57ca-4cbe-bec0-77fc915ca663 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:55.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d222ee2a-57ca-4cbe-bec0-77fc915ca663 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:55.623Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.624Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.624Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3d2207b-2ec6-4c93-a98b-40d5d4ce7166 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:55.624Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3d2207b-2ec6-4c93-a98b-40d5d4ce7166 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28968e4b 2021-05-04T01:29:55.624Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3d2207b-2ec6-4c93-a98b-40d5d4ce7166 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:55.624Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3d2207b-2ec6-4c93-a98b-40d5d4ce7166 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:55.628Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.701Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3934ce54-95da-4974-b1e8-0a8ae2d87279 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:55.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3934ce54-95da-4974-b1e8-0a8ae2d87279 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b096ffd 2021-05-04T01:29:55.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3934ce54-95da-4974-b1e8-0a8ae2d87279 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:55.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3934ce54-95da-4974-b1e8-0a8ae2d87279 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:55.706Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.708Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9eb1d4fd-f0da-45df-babc-06b3763cba34 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:55.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9eb1d4fd-f0da-45df-babc-06b3763cba34 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6286d16a 2021-05-04T01:29:55.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9eb1d4fd-f0da-45df-babc-06b3763cba34 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:55.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9eb1d4fd-f0da-45df-babc-06b3763cba34 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:55.714Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.716Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:55.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5290afc7-c460-4ae8-8fcb-89734ac760ee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:55.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5290afc7-c460-4ae8-8fcb-89734ac760ee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6924e590 2021-05-04T01:29:55.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5290afc7-c460-4ae8-8fcb-89734ac760ee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:55.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5290afc7-c460-4ae8-8fcb-89734ac760ee PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:29:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:29:58.661Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:58.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:29:58.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95800ac3-6a9f-4118-924a-f71a566e6ea7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:29:58.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95800ac3-6a9f-4118-924a-f71a566e6ea7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@755cda29 2021-05-04T01:29:58.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95800ac3-6a9f-4118-924a-f71a566e6ea7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:29:58.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95800ac3-6a9f-4118-924a-f71a566e6ea7 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:30:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:01.702Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:01.705Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:01.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58c865f4-9d15-4c8a-80f2-07fcfa23168d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:01.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58c865f4-9d15-4c8a-80f2-07fcfa23168d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75aa35f8 2021-05-04T01:30:01.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58c865f4-9d15-4c8a-80f2-07fcfa23168d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:01.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58c865f4-9d15-4c8a-80f2-07fcfa23168d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:30:03.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:04.701Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:30:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:05.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:30:05.185Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:30:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:05.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:05.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:05.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:05.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:05.190Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:30:05.282Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:05.282Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0626a269-6da6-4576-982d-1168859f233f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:05.282Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0626a269-6da6-4576-982d-1168859f233f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21f3b7a6 2021-05-04T01:30:05.282Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0626a269-6da6-4576-982d-1168859f233f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:05.282Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0626a269-6da6-4576-982d-1168859f233f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:30:05.285Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0626a269-6da6-4576-982d-1168859f233f ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:30:05.285Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=0626a269-6da6-4576-982d-1168859f233f ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:30:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:07.664Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:08.078Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0hlw7xgvgjpyh5s4d360nq9dt312 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01qbzxywnpo4kc1pwukt20wtcx4311 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01inils532r2pn1sp2qtfyf0dmm309 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0nvtl0ximzox11s1e2r4hjpjtt316 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01bm75igopo14e8v5z25y20sb1310 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node018qbwg8qnnnzon97mad8vwygf318 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01l06rilibtcon1aqhm6uvs08vz314 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node04t4ye5yr4uvp13lfaad171af7320 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ncuu8xet2n8wemnaiyd3eruh313 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0p9dibjz1ii6rry3lj14axmy8319 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01mjksjrrcdaq1pivqawyrzrnq317 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node04v7ihr3n3i91fz13snnha1v1308 2021-05-04T01:30:08.079Z [Scheduler-712609105-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01srn7eya9t8rblcwlimdxkdq315 2021-05-04T01:30:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:11.849Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:832) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:30:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:13.301Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:13.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a052101-8b97-419e-903e-fccdf054fadc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:13.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a052101-8b97-419e-903e-fccdf054fadc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6fb5de6d 2021-05-04T01:30:13.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a052101-8b97-419e-903e-fccdf054fadc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:13.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a052101-8b97-419e-903e-fccdf054fadc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:30:13.309Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:14.407Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:14.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=786090bf-468e-40d0-ad84-ccd2eb21709c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:14.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=786090bf-468e-40d0-ad84-ccd2eb21709c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c2bcc20 2021-05-04T01:30:14.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=786090bf-468e-40d0-ad84-ccd2eb21709c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:14.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=786090bf-468e-40d0-ad84-ccd2eb21709c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:30:14.427Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:14.826Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:30:14.828Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 256000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:15.100Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:15.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95488c2b-660c-4358-b386-2399afe62a2e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:15.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95488c2b-660c-4358-b386-2399afe62a2e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@580a33a8 2021-05-04T01:30:15.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95488c2b-660c-4358-b386-2399afe62a2e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:15.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95488c2b-660c-4358-b386-2399afe62a2e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:30:15.205Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:16.703Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:17.259Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:17.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb7c9d4b-43c7-4437-a597-6c83171aa5c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:17.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb7c9d4b-43c7-4437-a597-6c83171aa5c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64f8f188 2021-05-04T01:30:17.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb7c9d4b-43c7-4437-a597-6c83171aa5c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:17.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb7c9d4b-43c7-4437-a597-6c83171aa5c9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:30:17.680Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:85) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:30:17.681Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:30:00Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-05-04T01:30:17.680Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:832) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-05-04T01:30:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:19.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:30:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:20.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:30:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:30:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:20.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:30:20.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:20.187Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:30:20.954Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:20.955Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44f58275-6526-4f90-ac90-e8c8af821cfb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:20.955Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44f58275-6526-4f90-ac90-e8c8af821cfb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@257f1dcc 2021-05-04T01:30:20.955Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44f58275-6526-4f90-ac90-e8c8af821cfb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:20.955Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44f58275-6526-4f90-ac90-e8c8af821cfb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:30:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:22.700Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:23.048Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:832) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:30:23.547Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:85) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:30:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:26.956Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e8ac572-6d47-4d1c-b608-02046ef6272c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e8ac572-6d47-4d1c-b608-02046ef6272c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7179f8f 2021-05-04T01:30:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e8ac572-6d47-4d1c-b608-02046ef6272c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e8ac572-6d47-4d1c-b608-02046ef6272c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:30:26.967Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:30:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:35.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:30:35.184Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:30:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:35.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:35.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:35.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:35.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:35.190Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:30:36.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:30:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:30:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:30:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:30:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:30:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:30:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:30:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:42.546Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.8.171:9042: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:42.600Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:30:42.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dff3a76-f704-499e-8979-1eeef3681c21 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:42.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dff3a76-f704-499e-8979-1eeef3681c21 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e897bb6 2021-05-04T01:30:42.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dff3a76-f704-499e-8979-1eeef3681c21 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:42.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dff3a76-f704-499e-8979-1eeef3681c21 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:30:42.611Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9dff3a76-f704-499e-8979-1eeef3681c21 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:30:42.611Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=9dff3a76-f704-499e-8979-1eeef3681c21 ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:30:45.214Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:45.252Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:85) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-05-04T01:30:45.252Z [ApplicationDataTypeCacheThread-1] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Internal Connection problem. Description: Failed to fetch data types from graph(cache) 2021-05-04T01:30:45.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c5687c7-e691-4737-9113-1c73952781e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:45.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c5687c7-e691-4737-9113-1c73952781e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ddebf5e 2021-05-04T01:30:45.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c5687c7-e691-4737-9113-1c73952781e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:45.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c5687c7-e691-4737-9113-1c73952781e9 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:30:47.653Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=59052ec5-a829-4220-a9a3-478f45449f5d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:47.654Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=59052ec5-a829-4220-a9a3-478f45449f5d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70925654 2021-05-04T01:30:47.654Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=59052ec5-a829-4220-a9a3-478f45449f5d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:47.654Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=59052ec5-a829-4220-a9a3-478f45449f5d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:30:47.947Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0facb102-86d1-48b1-b2ae-59eb5b6b3960 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:47.947Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0facb102-86d1-48b1-b2ae-59eb5b6b3960 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4bc2442e 2021-05-04T01:30:47.947Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0facb102-86d1-48b1-b2ae-59eb5b6b3960 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:47.947Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0facb102-86d1-48b1-b2ae-59eb5b6b3960 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:30:48.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b76d6ab8-5c3f-41c6-96fd-12272f4a20cc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:48.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b76d6ab8-5c3f-41c6-96fd-12272f4a20cc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ed3d22b 2021-05-04T01:30:48.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b76d6ab8-5c3f-41c6-96fd-12272f4a20cc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:48.148Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b76d6ab8-5c3f-41c6-96fd-12272f4a20cc PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:30:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:48.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4633ccbb-d73d-4eb5-90ab-4ad5dcd96fd3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:48.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4633ccbb-d73d-4eb5-90ab-4ad5dcd96fd3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@562fb8e0 2021-05-04T01:30:48.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4633ccbb-d73d-4eb5-90ab-4ad5dcd96fd3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:48.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4633ccbb-d73d-4eb5-90ab-4ad5dcd96fd3 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:30:48.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a9dced7-b174-4a84-8862-b44300f21fd0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:48.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a9dced7-b174-4a84-8862-b44300f21fd0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@346d7487 2021-05-04T01:30:48.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a9dced7-b174-4a84-8862-b44300f21fd0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:48.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a9dced7-b174-4a84-8862-b44300f21fd0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:30:48.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=682f57a3-716f-48c0-b7ff-0c78c04e8abc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:48.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=682f57a3-716f-48c0-b7ff-0c78c04e8abc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6dd7b510 2021-05-04T01:30:48.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=682f57a3-716f-48c0-b7ff-0c78c04e8abc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:48.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=682f57a3-716f-48c0-b7ff-0c78c04e8abc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:30:48.763Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=682f57a3-716f-48c0-b7ff-0c78c04e8abc ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:30:48.763Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=682f57a3-716f-48c0-b7ff-0c78c04e8abc ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:30:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:30:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:50.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:30:50.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:30:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:30:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:30:50.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:30:50.264Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:826) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:30:50.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=484e7571-20a3-4bf3-b8fb-bbcc78dc4f7b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:30:50.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=484e7571-20a3-4bf3-b8fb-bbcc78dc4f7b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7d05600b 2021-05-04T01:30:50.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=484e7571-20a3-4bf3-b8fb-bbcc78dc4f7b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:30:50.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=484e7571-20a3-4bf3-b8fb-bbcc78dc4f7b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:30:50.450Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:30:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:54.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:30:57.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:31:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:31:02.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c233b50c-8d4a-4411-97bf-a327d0ecb022 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:02.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c233b50c-8d4a-4411-97bf-a327d0ecb022 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56463ff3 2021-05-04T01:31:02.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c233b50c-8d4a-4411-97bf-a327d0ecb022 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:02.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c233b50c-8d4a-4411-97bf-a327d0ecb022 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:02.453Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:30:47.158292Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:31:03.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:31:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:31:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:05.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:31:05.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:31:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:05.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:31:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:31:09.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1501d0f4-e4ef-4d5f-9563-49063a936d9f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:09.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1501d0f4-e4ef-4d5f-9563-49063a936d9f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@950a0f0 2021-05-04T01:31:09.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1501d0f4-e4ef-4d5f-9563-49063a936d9f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:09.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1501d0f4-e4ef-4d5f-9563-49063a936d9f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:09.460Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:31:10.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a934d56-648f-4a59-8fab-27890042b11b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:10.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a934d56-648f-4a59-8fab-27890042b11b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@597febff 2021-05-04T01:31:10.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a934d56-648f-4a59-8fab-27890042b11b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:10.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a934d56-648f-4a59-8fab-27890042b11b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:11.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d41c65c-2526-4116-877c-e39b5d095e40 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:11.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d41c65c-2526-4116-877c-e39b5d095e40 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f2cdde3 2021-05-04T01:31:11.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d41c65c-2526-4116-877c-e39b5d095e40 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:11.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d41c65c-2526-4116-877c-e39b5d095e40 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:11.165Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7d41c65c-2526-4116-877c-e39b5d095e40 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:31:11.165Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7d41c65c-2526-4116-877c-e39b5d095e40 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:31:12.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa76c534-7757-4e9c-856c-be031202faf6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:12.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa76c534-7757-4e9c-856c-be031202faf6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6872c252 2021-05-04T01:31:12.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa76c534-7757-4e9c-856c-be031202faf6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:12.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa76c534-7757-4e9c-856c-be031202faf6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:13.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5012b8e1-54f1-498f-97ae-69dabe7ef4ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:13.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5012b8e1-54f1-498f-97ae-69dabe7ef4ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64a9a88c 2021-05-04T01:31:13.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5012b8e1-54f1-498f-97ae-69dabe7ef4ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:13.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5012b8e1-54f1-498f-97ae-69dabe7ef4ff PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:14.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26b664ca-3f6c-4296-a9c5-18d1c1d5d057 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:14.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26b664ca-3f6c-4296-a9c5-18d1c1d5d057 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@422f97f5 2021-05-04T01:31:14.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26b664ca-3f6c-4296-a9c5-18d1c1d5d057 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:14.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26b664ca-3f6c-4296-a9c5-18d1c1d5d057 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:15.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=064e9b52-3a95-477a-927a-55ca4560e85b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:15.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=064e9b52-3a95-477a-927a-55ca4560e85b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b858996 2021-05-04T01:31:15.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=064e9b52-3a95-477a-927a-55ca4560e85b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:15.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=064e9b52-3a95-477a-927a-55ca4560e85b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:17.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99c69aa1-8819-4e72-b280-40c22e0a6420 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:17.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99c69aa1-8819-4e72-b280-40c22e0a6420 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@559baee6 2021-05-04T01:31:17.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99c69aa1-8819-4e72-b280-40c22e0a6420 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:17.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99c69aa1-8819-4e72-b280-40c22e0a6420 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:31:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:20.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:31:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:31:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:20.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:31:20.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec0018cc-e891-408e-a20b-d6cfc2838e23 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:20.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec0018cc-e891-408e-a20b-d6cfc2838e23 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c5f9111 2021-05-04T01:31:20.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec0018cc-e891-408e-a20b-d6cfc2838e23 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:20.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec0018cc-e891-408e-a20b-d6cfc2838e23 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=beb11ea8-0fd9-447c-afe6-47ff3992e1db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=beb11ea8-0fd9-447c-afe6-47ff3992e1db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42712150 2021-05-04T01:31:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=beb11ea8-0fd9-447c-afe6-47ff3992e1db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:23.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=beb11ea8-0fd9-447c-afe6-47ff3992e1db PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:26.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=995717ea-cfb0-4e2a-86aa-953cd3b70ab9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:26.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=995717ea-cfb0-4e2a-86aa-953cd3b70ab9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@453fd9b1 2021-05-04T01:31:26.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=995717ea-cfb0-4e2a-86aa-953cd3b70ab9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:26.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=995717ea-cfb0-4e2a-86aa-953cd3b70ab9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:29.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec4c19ad-425a-429b-9e36-0e3a3dd3abb5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:29.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec4c19ad-425a-429b-9e36-0e3a3dd3abb5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ddbb36e 2021-05-04T01:31:29.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec4c19ad-425a-429b-9e36-0e3a3dd3abb5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:29.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec4c19ad-425a-429b-9e36-0e3a3dd3abb5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c93156df-fa43-45e7-a58a-0bfda6038fe8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c93156df-fa43-45e7-a58a-0bfda6038fe8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d0c64dd 2021-05-04T01:31:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c93156df-fa43-45e7-a58a-0bfda6038fe8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:32.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c93156df-fa43-45e7-a58a-0bfda6038fe8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:31:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:35.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:35.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:35.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:31:35.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:31:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:35.186Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:31:35.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=476bd440-8602-482f-ab8e-df9f43017cd1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:35.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=476bd440-8602-482f-ab8e-df9f43017cd1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6bca78d 2021-05-04T01:31:35.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=476bd440-8602-482f-ab8e-df9f43017cd1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:35.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=476bd440-8602-482f-ab8e-df9f43017cd1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:31:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:31:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:31:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:31:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:31:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:31:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:31:38.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e19cc8a4-4371-4633-acf0-0905e29c8772 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:38.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e19cc8a4-4371-4633-acf0-0905e29c8772 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d7504ba 2021-05-04T01:31:38.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e19cc8a4-4371-4633-acf0-0905e29c8772 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:38.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e19cc8a4-4371-4633-acf0-0905e29c8772 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:41.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b9d5905-9bd3-4edd-960c-120aace563fb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:41.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b9d5905-9bd3-4edd-960c-120aace563fb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d2580b9 2021-05-04T01:31:41.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b9d5905-9bd3-4edd-960c-120aace563fb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:41.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b9d5905-9bd3-4edd-960c-120aace563fb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e8f8a6b-713c-4cc5-8607-02f7dd1cf49c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e8f8a6b-713c-4cc5-8607-02f7dd1cf49c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a1234c 2021-05-04T01:31:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e8f8a6b-713c-4cc5-8607-02f7dd1cf49c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:44.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e8f8a6b-713c-4cc5-8607-02f7dd1cf49c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:46.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3fa3531-36e2-48f6-926a-d0af10924aa5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:46.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3fa3531-36e2-48f6-926a-d0af10924aa5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60c27c7a 2021-05-04T01:31:46.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3fa3531-36e2-48f6-926a-d0af10924aa5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:46.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3fa3531-36e2-48f6-926a-d0af10924aa5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:31:50.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:50.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:50.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:31:50.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:31:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:31:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:31:50.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:31:50.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d16964ec-c5b5-404e-9b5b-100bb766f819 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:50.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d16964ec-c5b5-404e-9b5b-100bb766f819 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7507f40c 2021-05-04T01:31:50.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d16964ec-c5b5-404e-9b5b-100bb766f819 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:50.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d16964ec-c5b5-404e-9b5b-100bb766f819 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:53.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7ca5268-6903-4df0-8221-f99924849e15 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:53.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7ca5268-6903-4df0-8221-f99924849e15 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7426825d 2021-05-04T01:31:53.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7ca5268-6903-4df0-8221-f99924849e15 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:53.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7ca5268-6903-4df0-8221-f99924849e15 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:56.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb0edc7b-9d79-4f22-b02d-f00727269914 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:56.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb0edc7b-9d79-4f22-b02d-f00727269914 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e1764d6 2021-05-04T01:31:56.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb0edc7b-9d79-4f22-b02d-f00727269914 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:56.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb0edc7b-9d79-4f22-b02d-f00727269914 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:31:59.152Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e30d8a5c-ce34-4c25-93c4-cc35e81ef3af PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:31:59.152Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e30d8a5c-ce34-4c25-93c4-cc35e81ef3af PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d45e6c2 2021-05-04T01:31:59.152Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e30d8a5c-ce34-4c25-93c4-cc35e81ef3af PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:31:59.152Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e30d8a5c-ce34-4c25-93c4-cc35e81ef3af PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:02.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b521c7c-5fca-4fe6-a3d0-3ddfb9e712fc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:02.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b521c7c-5fca-4fe6-a3d0-3ddfb9e712fc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4728d4f2 2021-05-04T01:32:02.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b521c7c-5fca-4fe6-a3d0-3ddfb9e712fc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:02.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b521c7c-5fca-4fe6-a3d0-3ddfb9e712fc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:32:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:05.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:32:05.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:32:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:05.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:32:05.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f011b57-514c-4949-93f7-412fc7315e9d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:05.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f011b57-514c-4949-93f7-412fc7315e9d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7dc9220 2021-05-04T01:32:05.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f011b57-514c-4949-93f7-412fc7315e9d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:05.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f011b57-514c-4949-93f7-412fc7315e9d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:08.747Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84b5092b-6136-47f7-9ee1-b40a0526665b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:08.747Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84b5092b-6136-47f7-9ee1-b40a0526665b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@422877d9 2021-05-04T01:32:08.747Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84b5092b-6136-47f7-9ee1-b40a0526665b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:08.747Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=84b5092b-6136-47f7-9ee1-b40a0526665b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:11.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e55b3ee3-7710-4ebf-8e14-46928c166d56 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:11.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e55b3ee3-7710-4ebf-8e14-46928c166d56 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6079eaa8 2021-05-04T01:32:11.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e55b3ee3-7710-4ebf-8e14-46928c166d56 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:11.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e55b3ee3-7710-4ebf-8e14-46928c166d56 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:17.470Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=197ff4cb-3796-48e7-8370-4e40ea6310d7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:17.470Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=197ff4cb-3796-48e7-8370-4e40ea6310d7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61773d11 2021-05-04T01:32:17.470Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=197ff4cb-3796-48e7-8370-4e40ea6310d7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:17.470Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=197ff4cb-3796-48e7-8370-4e40ea6310d7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:18.954Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9b54029-d3dc-42cb-b7f3-1436d3d75677 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:18.954Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9b54029-d3dc-42cb-b7f3-1436d3d75677 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d9ee8d 2021-05-04T01:32:18.954Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9b54029-d3dc-42cb-b7f3-1436d3d75677 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:18.954Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9b54029-d3dc-42cb-b7f3-1436d3d75677 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:32:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:20.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:32:20.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:32:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:20.181Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:32:20.654Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc779d7c-ae68-400a-bf9c-db710c6096cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:20.654Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc779d7c-ae68-400a-bf9c-db710c6096cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ced7b0 2021-05-04T01:32:20.655Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc779d7c-ae68-400a-bf9c-db710c6096cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:20.655Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc779d7c-ae68-400a-bf9c-db710c6096cf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9df92c8-85d7-4638-9893-083aa3b51c0a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9df92c8-85d7-4638-9893-083aa3b51c0a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ccd51cb 2021-05-04T01:32:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9df92c8-85d7-4638-9893-083aa3b51c0a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:23.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9df92c8-85d7-4638-9893-083aa3b51c0a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:28.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b835ede8-d994-443e-9602-218f09f269d4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:28.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b835ede8-d994-443e-9602-218f09f269d4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@748a7ef6 2021-05-04T01:32:28.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b835ede8-d994-443e-9602-218f09f269d4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:28.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b835ede8-d994-443e-9602-218f09f269d4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:30.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08a4800c-2b7c-41c2-b17e-64ea64db6975 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:30.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08a4800c-2b7c-41c2-b17e-64ea64db6975 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@396a4b6c 2021-05-04T01:32:30.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08a4800c-2b7c-41c2-b17e-64ea64db6975 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:30.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08a4800c-2b7c-41c2-b17e-64ea64db6975 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:32.747Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ecbc618-519b-454f-be19-71ea98e820de PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:32.747Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ecbc618-519b-454f-be19-71ea98e820de PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c6c78a6 2021-05-04T01:32:32.747Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ecbc618-519b-454f-be19-71ea98e820de PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:32.747Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ecbc618-519b-454f-be19-71ea98e820de PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:32:35.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:35.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:35.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:32:35.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:32:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:35.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:35.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:35.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:35.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:35.191Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:32:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92ecdfab-647c-4062-818c-1d352c3535ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92ecdfab-647c-4062-818c-1d352c3535ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a692be9 2021-05-04T01:32:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92ecdfab-647c-4062-818c-1d352c3535ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:35.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92ecdfab-647c-4062-818c-1d352c3535ca PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:32:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:32:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:32:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:32:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:37.113Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:32:37.113Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:32:37.114Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:32:40.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bab073e7-dd8f-45c4-be77-52d103b43ce1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:40.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bab073e7-dd8f-45c4-be77-52d103b43ce1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2db3899c 2021-05-04T01:32:40.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bab073e7-dd8f-45c4-be77-52d103b43ce1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:40.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bab073e7-dd8f-45c4-be77-52d103b43ce1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:41.353Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35e00924-0a54-4530-869c-c56ad3b7f2a1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:41.353Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35e00924-0a54-4530-869c-c56ad3b7f2a1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4669966d 2021-05-04T01:32:41.353Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35e00924-0a54-4530-869c-c56ad3b7f2a1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:41.353Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35e00924-0a54-4530-869c-c56ad3b7f2a1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:44.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b79dbd27-fd0b-42db-b40d-011d34a6cc2f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:44.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b79dbd27-fd0b-42db-b40d-011d34a6cc2f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@696f6f4e 2021-05-04T01:32:44.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b79dbd27-fd0b-42db-b40d-011d34a6cc2f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:44.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b79dbd27-fd0b-42db-b40d-011d34a6cc2f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:46.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0ac65b9-b2e3-4b56-b4bd-504c9b6af346 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:46.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0ac65b9-b2e3-4b56-b4bd-504c9b6af346 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a9a37e9 2021-05-04T01:32:46.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0ac65b9-b2e3-4b56-b4bd-504c9b6af346 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:46.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0ac65b9-b2e3-4b56-b4bd-504c9b6af346 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:49.847Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5154e994-1c68-42e7-afcf-4656b26b200b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:49.847Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5154e994-1c68-42e7-afcf-4656b26b200b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bd4b44f 2021-05-04T01:32:49.847Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5154e994-1c68-42e7-afcf-4656b26b200b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:49.847Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5154e994-1c68-42e7-afcf-4656b26b200b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:50.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:32:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:32:50.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:50.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:32:50.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:32:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:32:50.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:32:50.186Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:32:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98cbe878-e2d8-48b6-94b7-1aa8b6d94b88 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98cbe878-e2d8-48b6-94b7-1aa8b6d94b88 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@118493a6 2021-05-04T01:32:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98cbe878-e2d8-48b6-94b7-1aa8b6d94b88 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:53.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98cbe878-e2d8-48b6-94b7-1aa8b6d94b88 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:56.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d84f95d3-4ae1-48bc-8963-a6a575091298 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:56.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d84f95d3-4ae1-48bc-8963-a6a575091298 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3838bc7 2021-05-04T01:32:56.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d84f95d3-4ae1-48bc-8963-a6a575091298 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:56.201Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d84f95d3-4ae1-48bc-8963-a6a575091298 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:32:59.785Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a734cce-0255-4a68-a624-25be9635fc6b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:32:59.785Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a734cce-0255-4a68-a624-25be9635fc6b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78ddc121 2021-05-04T01:32:59.785Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a734cce-0255-4a68-a624-25be9635fc6b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:32:59.785Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a734cce-0255-4a68-a624-25be9635fc6b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:02.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bd2beb2-5f43-451e-9dfb-ac16c971f9ed PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:02.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bd2beb2-5f43-451e-9dfb-ac16c971f9ed PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@461564b9 2021-05-04T01:33:02.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bd2beb2-5f43-451e-9dfb-ac16c971f9ed PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:02.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1bd2beb2-5f43-451e-9dfb-ac16c971f9ed PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:05.172Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:33:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:05.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:33:05.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:33:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:33:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:05.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:05.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:05.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:05.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:05.192Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:33:05.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da843982-b8c1-4360-8fae-1d314b7b37ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:05.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da843982-b8c1-4360-8fae-1d314b7b37ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73b9d67e 2021-05-04T01:33:05.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da843982-b8c1-4360-8fae-1d314b7b37ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:05.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da843982-b8c1-4360-8fae-1d314b7b37ca PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:08.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ad8a5aa-1625-4952-a6dd-eb5d8d6fb5ef PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:08.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ad8a5aa-1625-4952-a6dd-eb5d8d6fb5ef PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6cfba67d 2021-05-04T01:33:08.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ad8a5aa-1625-4952-a6dd-eb5d8d6fb5ef PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:08.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ad8a5aa-1625-4952-a6dd-eb5d8d6fb5ef PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:11.154Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bc11857-3011-4dc8-a8b1-05f817360b76 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:11.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bc11857-3011-4dc8-a8b1-05f817360b76 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a39b60a 2021-05-04T01:33:11.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bc11857-3011-4dc8-a8b1-05f817360b76 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:11.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3bc11857-3011-4dc8-a8b1-05f817360b76 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:14.987Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a2f3a9b-972c-4891-ab83-dc4e9815ec67 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:14.987Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a2f3a9b-972c-4891-ab83-dc4e9815ec67 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17e42558 2021-05-04T01:33:14.987Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a2f3a9b-972c-4891-ab83-dc4e9815ec67 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:14.988Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a2f3a9b-972c-4891-ab83-dc4e9815ec67 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:16.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7741e295-c2eb-4352-84a1-44081bbdff62 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:16.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7741e295-c2eb-4352-84a1-44081bbdff62 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ad31bfc 2021-05-04T01:33:16.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7741e295-c2eb-4352-84a1-44081bbdff62 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:16.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7741e295-c2eb-4352-84a1-44081bbdff62 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:20.054Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a7fd82f-6fc2-4ca4-874b-5f3c52aacb27 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:20.054Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a7fd82f-6fc2-4ca4-874b-5f3c52aacb27 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1baa8895 2021-05-04T01:33:20.054Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a7fd82f-6fc2-4ca4-874b-5f3c52aacb27 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:20.054Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a7fd82f-6fc2-4ca4-874b-5f3c52aacb27 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:33:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:20.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:33:20.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:33:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:20.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:33:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:20.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:33:23.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e24575e-b29c-4e50-b7ad-77003b3caca7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:23.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e24575e-b29c-4e50-b7ad-77003b3caca7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c94e40c 2021-05-04T01:33:23.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e24575e-b29c-4e50-b7ad-77003b3caca7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:23.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7e24575e-b29c-4e50-b7ad-77003b3caca7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:26.726Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10ab800d-f42e-4267-b905-2c9511961197 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:26.726Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10ab800d-f42e-4267-b905-2c9511961197 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1aecda23 2021-05-04T01:33:26.726Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10ab800d-f42e-4267-b905-2c9511961197 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:26.726Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10ab800d-f42e-4267-b905-2c9511961197 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:29.012Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f18b901-9f83-4a10-b1af-46f23e4f25f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:29.012Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f18b901-9f83-4a10-b1af-46f23e4f25f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28d22e79 2021-05-04T01:33:29.012Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f18b901-9f83-4a10-b1af-46f23e4f25f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:29.013Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f18b901-9f83-4a10-b1af-46f23e4f25f5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:32.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7aeb5c7-ed8f-46ca-87a8-e1fea9b58492 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:32.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7aeb5c7-ed8f-46ca-87a8-e1fea9b58492 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6773e5d6 2021-05-04T01:33:32.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7aeb5c7-ed8f-46ca-87a8-e1fea9b58492 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:32.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7aeb5c7-ed8f-46ca-87a8-e1fea9b58492 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:32.367Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=e7aeb5c7-ed8f-46ca-87a8-e1fea9b58492 PartnerName=SDC.BE I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://sdc-onboarding-be.onap:8445: The target server failed to respond 2021-05-04T01:33:32.367Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=e7aeb5c7-ed8f-46ca-87a8-e1fea9b58492 PartnerName=SDC.BE Retrying request to {s}->https://sdc-onboarding-be.onap:8445 2021-05-04T01:33:32.382Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e7aeb5c7-ed8f-46ca-87a8-e1fea9b58492 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:33:32.383Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e7aeb5c7-ed8f-46ca-87a8-e1fea9b58492 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:33:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:33:35.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:35.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:35.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:33:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:33:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:35.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:33:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db3c61ca-4ee6-4cfa-968d-6d9e4d7438af PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db3c61ca-4ee6-4cfa-968d-6d9e4d7438af PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@108d2bc8 2021-05-04T01:33:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db3c61ca-4ee6-4cfa-968d-6d9e4d7438af PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db3c61ca-4ee6-4cfa-968d-6d9e4d7438af PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:35.504Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=db3c61ca-4ee6-4cfa-968d-6d9e4d7438af ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:33:35.504Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=db3c61ca-4ee6-4cfa-968d-6d9e4d7438af ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:33:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:33:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:33:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:33:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:33:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:37.116Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:33:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:33:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:33:38.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=accac34b-fc2b-489d-9f02-01d9649f4b5c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:38.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=accac34b-fc2b-489d-9f02-01d9649f4b5c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@513073f7 2021-05-04T01:33:38.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=accac34b-fc2b-489d-9f02-01d9649f4b5c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:38.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=accac34b-fc2b-489d-9f02-01d9649f4b5c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:41.270Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=accac34b-fc2b-489d-9f02-01d9649f4b5c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:33:41.270Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=accac34b-fc2b-489d-9f02-01d9649f4b5c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:33:42.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2265d0e-d2f9-4420-ad09-2f3c9c48fcf3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:42.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2265d0e-d2f9-4420-ad09-2f3c9c48fcf3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@229192fc 2021-05-04T01:33:42.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2265d0e-d2f9-4420-ad09-2f3c9c48fcf3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:42.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2265d0e-d2f9-4420-ad09-2f3c9c48fcf3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:45.211Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c2265d0e-d2f9-4420-ad09-2f3c9c48fcf3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:33:45.211Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c2265d0e-d2f9-4420-ad09-2f3c9c48fcf3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:33:46.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c19f30a-3eef-4a87-8513-5b0f618f22a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:46.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c19f30a-3eef-4a87-8513-5b0f618f22a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72094f84 2021-05-04T01:33:46.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c19f30a-3eef-4a87-8513-5b0f618f22a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:46.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c19f30a-3eef-4a87-8513-5b0f618f22a3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:49.215Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4c19f30a-3eef-4a87-8513-5b0f618f22a3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:33:49.216Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4c19f30a-3eef-4a87-8513-5b0f618f22a3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:33:49.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fc487af-08cb-4724-82d4-1f2728fe5b39 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:49.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fc487af-08cb-4724-82d4-1f2728fe5b39 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e11f885 2021-05-04T01:33:49.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fc487af-08cb-4724-82d4-1f2728fe5b39 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:49.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fc487af-08cb-4724-82d4-1f2728fe5b39 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:33:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:50.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:33:50.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:33:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:50.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:33:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:33:50.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:33:52.877Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0fc487af-08cb-4724-82d4-1f2728fe5b39 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:33:52.877Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0fc487af-08cb-4724-82d4-1f2728fe5b39 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:33:53.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=935ff511-8b6d-466d-974e-69131514f7a5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:53.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=935ff511-8b6d-466d-974e-69131514f7a5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67546476 2021-05-04T01:33:53.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=935ff511-8b6d-466d-974e-69131514f7a5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:53.709Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=935ff511-8b6d-466d-974e-69131514f7a5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:33:56.726Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=935ff511-8b6d-466d-974e-69131514f7a5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:33:56.726Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=935ff511-8b6d-466d-974e-69131514f7a5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:33:57.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1004bde2-7078-4cdd-b330-779276fdf098 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:33:57.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1004bde2-7078-4cdd-b330-779276fdf098 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ac5bddf 2021-05-04T01:33:57.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1004bde2-7078-4cdd-b330-779276fdf098 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:33:57.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1004bde2-7078-4cdd-b330-779276fdf098 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:34:00.412Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=1004bde2-7078-4cdd-b330-779276fdf098 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:34:00.412Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=1004bde2-7078-4cdd-b330-779276fdf098 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:34:01.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d75002e-eea0-430f-956a-82a876cfd952 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:01.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d75002e-eea0-430f-956a-82a876cfd952 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a503b82 2021-05-04T01:34:01.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d75002e-eea0-430f-956a-82a876cfd952 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:01.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d75002e-eea0-430f-956a-82a876cfd952 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:34:04.254Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5d75002e-eea0-430f-956a-82a876cfd952 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:04.254Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5d75002e-eea0-430f-956a-82a876cfd952 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:05.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c707ba5-db7e-470c-93c8-f01a9f6f19ce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:05.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c707ba5-db7e-470c-93c8-f01a9f6f19ce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2bd42288 2021-05-04T01:34:05.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c707ba5-db7e-470c-93c8-f01a9f6f19ce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:05.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c707ba5-db7e-470c-93c8-f01a9f6f19ce PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:34:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:34:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:05.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:34:05.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:34:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:05.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:34:05.660Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7c707ba5-db7e-470c-93c8-f01a9f6f19ce ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:34:05.661Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7c707ba5-db7e-470c-93c8-f01a9f6f19ce ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:34:06.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3963755-c9de-4193-a737-e58d7a1a5b77 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:06.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3963755-c9de-4193-a737-e58d7a1a5b77 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72bd4271 2021-05-04T01:34:06.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3963755-c9de-4193-a737-e58d7a1a5b77 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:06.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3963755-c9de-4193-a737-e58d7a1a5b77 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:34:07.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=515d1894-fa23-4430-99b2-f707c662843b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:07.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=515d1894-fa23-4430-99b2-f707c662843b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3cc76dba 2021-05-04T01:34:07.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=515d1894-fa23-4430-99b2-f707c662843b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:07.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=515d1894-fa23-4430-99b2-f707c662843b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:34:07.876Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5752d0fe-f725-4147-85d0-b1f8baf11344 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:07.876Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5752d0fe-f725-4147-85d0-b1f8baf11344 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@360dc8c8 2021-05-04T01:34:07.877Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5752d0fe-f725-4147-85d0-b1f8baf11344 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:07.877Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5752d0fe-f725-4147-85d0-b1f8baf11344 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:34:10.883Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5752d0fe-f725-4147-85d0-b1f8baf11344 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:10.884Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5752d0fe-f725-4147-85d0-b1f8baf11344 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:10.884Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5752d0fe-f725-4147-85d0-b1f8baf11344 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:34:10.884Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5752d0fe-f725-4147-85d0-b1f8baf11344 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:34:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:34:15.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:15.904Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:34:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:20.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:34:20.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:34:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:20.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:34:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:24.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:30.834Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:34:30.837Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 512000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:30.926Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:30.927Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:30.933Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:30.933Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9135fc62-a76b-46da-8a1c-945184d1cd29 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:30.933Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9135fc62-a76b-46da-8a1c-945184d1cd29 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@481eebb4 2021-05-04T01:34:30.933Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9135fc62-a76b-46da-8a1c-945184d1cd29 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:30.933Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9135fc62-a76b-46da-8a1c-945184d1cd29 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:34:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:33.983Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=9135fc62-a76b-46da-8a1c-945184d1cd29 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:33.983Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=9135fc62-a76b-46da-8a1c-945184d1cd29 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:33.984Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9135fc62-a76b-46da-8a1c-945184d1cd29 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:34:33.984Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=9135fc62-a76b-46da-8a1c-945184d1cd29 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:34:34.001Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:34.002Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:34.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=137e896d-ec21-431f-a30b-4fdfec0e8605 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:34.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=137e896d-ec21-431f-a30b-4fdfec0e8605 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57858b0e 2021-05-04T01:34:34.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=137e896d-ec21-431f-a30b-4fdfec0e8605 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:34.003Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=137e896d-ec21-431f-a30b-4fdfec0e8605 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:34:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:34:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:35.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:34:35.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:34:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:35.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:34:36.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:37.038Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=137e896d-ec21-431f-a30b-4fdfec0e8605 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:37.039Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=137e896d-ec21-431f-a30b-4fdfec0e8605 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:37.039Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=137e896d-ec21-431f-a30b-4fdfec0e8605 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:34:37.039Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=137e896d-ec21-431f-a30b-4fdfec0e8605 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:34:37.041Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:37.099Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:37.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8178f0b-ae41-47e2-9a9f-a372f80220b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:37.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8178f0b-ae41-47e2-9a9f-a372f80220b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@306a4bb3 2021-05-04T01:34:37.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8178f0b-ae41-47e2-9a9f-a372f80220b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:37.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8178f0b-ae41-47e2-9a9f-a372f80220b9 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:34:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:34:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:34:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:34:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:34:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:34:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:34:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:34:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:40.149Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f8178f0b-ae41-47e2-9a9f-a372f80220b9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:40.149Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f8178f0b-ae41-47e2-9a9f-a372f80220b9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:40.152Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:40.154Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:40.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26761ac-26d4-4a11-831c-27e5150392df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:40.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26761ac-26d4-4a11-831c-27e5150392df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5fc0813d 2021-05-04T01:34:40.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26761ac-26d4-4a11-831c-27e5150392df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:40.155Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26761ac-26d4-4a11-831c-27e5150392df PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:34:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:43.199Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c26761ac-26d4-4a11-831c-27e5150392df ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:43.199Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c26761ac-26d4-4a11-831c-27e5150392df ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:43.202Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:43.204Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:43.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6253f5f4-4b51-44c1-a3f3-3d4b1d23ee54 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:43.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6253f5f4-4b51-44c1-a3f3-3d4b1d23ee54 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@abc3e6 2021-05-04T01:34:43.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6253f5f4-4b51-44c1-a3f3-3d4b1d23ee54 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:43.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6253f5f4-4b51-44c1-a3f3-3d4b1d23ee54 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:34:45.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:46.244Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6253f5f4-4b51-44c1-a3f3-3d4b1d23ee54 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:46.245Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6253f5f4-4b51-44c1-a3f3-3d4b1d23ee54 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:46.247Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:46.248Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:46.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d61ee1c7-6f8f-4f2b-8d90-8b91a284a747 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:46.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d61ee1c7-6f8f-4f2b-8d90-8b91a284a747 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ad41ec8 2021-05-04T01:34:46.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d61ee1c7-6f8f-4f2b-8d90-8b91a284a747 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:46.248Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d61ee1c7-6f8f-4f2b-8d90-8b91a284a747 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:34:47.210Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:34:49.293Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d61ee1c7-6f8f-4f2b-8d90-8b91a284a747 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:49.293Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d61ee1c7-6f8f-4f2b-8d90-8b91a284a747 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:49.296Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:49.298Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:49.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d87be4b-2205-4902-b932-cf631146bb9b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:49.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d87be4b-2205-4902-b932-cf631146bb9b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70d0881 2021-05-04T01:34:49.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d87be4b-2205-4902-b932-cf631146bb9b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:49.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d87be4b-2205-4902-b932-cf631146bb9b PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:34:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:34:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:50.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:34:50.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:34:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:34:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:34:50.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:34:52.339Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7d87be4b-2205-4902-b932-cf631146bb9b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:52.339Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7d87be4b-2205-4902-b932-cf631146bb9b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:52.339Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7d87be4b-2205-4902-b932-cf631146bb9b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:34:52.339Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=7d87be4b-2205-4902-b932-cf631146bb9b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:34:52.342Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:52.344Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:52.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f593ebd-40e0-4eca-8cf3-a5d9d6afeb20 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:52.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f593ebd-40e0-4eca-8cf3-a5d9d6afeb20 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49c24f3f 2021-05-04T01:34:52.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f593ebd-40e0-4eca-8cf3-a5d9d6afeb20 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:52.344Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f593ebd-40e0-4eca-8cf3-a5d9d6afeb20 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:34:55.377Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6f593ebd-40e0-4eca-8cf3-a5d9d6afeb20 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:55.377Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6f593ebd-40e0-4eca-8cf3-a5d9d6afeb20 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:55.401Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:55.403Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:55.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8dd64c-413c-46ae-8870-4090e46b7f17 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:55.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8dd64c-413c-46ae-8870-4090e46b7f17 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4545cc66 2021-05-04T01:34:55.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8dd64c-413c-46ae-8870-4090e46b7f17 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:55.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c8dd64c-413c-46ae-8870-4090e46b7f17 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:34:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:34:57.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:34:58.435Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6c8dd64c-413c-46ae-8870-4090e46b7f17 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:34:58.435Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6c8dd64c-413c-46ae-8870-4090e46b7f17 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:34:58.440Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:58.442Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:34:58.442Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e5aa232-2236-4292-8f4e-a40a34e6180a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:34:58.442Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e5aa232-2236-4292-8f4e-a40a34e6180a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c638d99 2021-05-04T01:34:58.442Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e5aa232-2236-4292-8f4e-a40a34e6180a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:34:58.443Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e5aa232-2236-4292-8f4e-a40a34e6180a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:35:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:01.479Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2e5aa232-2236-4292-8f4e-a40a34e6180a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:35:01.479Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2e5aa232-2236-4292-8f4e-a40a34e6180a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:35:01.479Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2e5aa232-2236-4292-8f4e-a40a34e6180a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:35:01.479Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=2e5aa232-2236-4292-8f4e-a40a34e6180a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:35:01.500Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:01.503Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:01.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=691667e1-e8c7-4861-b29f-45b8e866b8d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:01.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=691667e1-e8c7-4861-b29f-45b8e866b8d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31fe6faa 2021-05-04T01:35:01.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=691667e1-e8c7-4861-b29f-45b8e866b8d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:01.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=691667e1-e8c7-4861-b29f-45b8e866b8d6 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:35:03.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:03.552Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:85) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:35:04.538Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=691667e1-e8c7-4861-b29f-45b8e866b8d6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:35:04.539Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=691667e1-e8c7-4861-b29f-45b8e866b8d6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:35:04.542Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:04.544Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:04.544Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a968d9d8-7594-40f5-a188-2b0d380de7a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:04.544Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a968d9d8-7594-40f5-a188-2b0d380de7a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a1613ea 2021-05-04T01:35:04.544Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a968d9d8-7594-40f5-a188-2b0d380de7a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:04.545Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a968d9d8-7594-40f5-a188-2b0d380de7a2 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:35:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:35:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:05.174Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:35:05.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:35:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:05.180Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:35:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:07.583Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a968d9d8-7594-40f5-a188-2b0d380de7a2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:35:07.583Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a968d9d8-7594-40f5-a188-2b0d380de7a2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:35:07.587Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:07.589Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:07.590Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c024317a-30fc-4748-aa17-0fe8baa4f16f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:07.590Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c024317a-30fc-4748-aa17-0fe8baa4f16f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6fd079e7 2021-05-04T01:35:07.590Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c024317a-30fc-4748-aa17-0fe8baa4f16f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:07.590Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c024317a-30fc-4748-aa17-0fe8baa4f16f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:35:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:10.642Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c024317a-30fc-4748-aa17-0fe8baa4f16f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:35:10.642Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c024317a-30fc-4748-aa17-0fe8baa4f16f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:35:10.645Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:10.648Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:10.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f8bcae-4a08-41c8-a1d4-eece03c6af50 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:10.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f8bcae-4a08-41c8-a1d4-eece03c6af50 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65e9013a 2021-05-04T01:35:10.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f8bcae-4a08-41c8-a1d4-eece03c6af50 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:10.648Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73f8bcae-4a08-41c8-a1d4-eece03c6af50 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:35:11.256Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:35:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:13.681Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=73f8bcae-4a08-41c8-a1d4-eece03c6af50 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:35:13.681Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=73f8bcae-4a08-41c8-a1d4-eece03c6af50 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:35:13.707Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:13.902Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:13.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9222549-c081-492a-aad4-f745ee4963c7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:13.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9222549-c081-492a-aad4-f745ee4963c7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a2e9c6a 2021-05-04T01:35:13.903Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9222549-c081-492a-aad4-f745ee4963c7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:13.903Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9222549-c081-492a-aad4-f745ee4963c7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:35:15.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:16.356Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:35:16.941Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f9222549-c081-492a-aad4-f745ee4963c7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:35:16.941Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f9222549-c081-492a-aad4-f745ee4963c7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:35:16.942Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f9222549-c081-492a-aad4-f745ee4963c7 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:35:16.942Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f9222549-c081-492a-aad4-f745ee4963c7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:35:17.000Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:19.303Z [cluster2-nio-worker-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:215) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:242) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:35:19.305Z [cluster2-nio-worker-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:215) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:242) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:35:19.763Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:19.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0220a3b4-b287-4713-8289-17238fbf7ad7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:19.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0220a3b4-b287-4713-8289-17238fbf7ad7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e53852f 2021-05-04T01:35:19.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0220a3b4-b287-4713-8289-17238fbf7ad7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:19.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0220a3b4-b287-4713-8289-17238fbf7ad7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:35:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:35:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:20.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:35:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:35:20.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:20.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:35:20.868Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:35:10.222650Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:35:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:22.801Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0220a3b4-b287-4713-8289-17238fbf7ad7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:35:22.801Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0220a3b4-b287-4713-8289-17238fbf7ad7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:35:22.804Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:24.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:26.471Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:35:27.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:35:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:35.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:35.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:35.179Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:35:35.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:35:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:35.186Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:35:36.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:35:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:35:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:35:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:35:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:37.116Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:35:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:35:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:35:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:42.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:45.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:46.529Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:46.534Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:46.535Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d871cbef-1bdd-4002-aefd-aef2aa123aa0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:46.535Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d871cbef-1bdd-4002-aefd-aef2aa123aa0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18e5ee78 2021-05-04T01:35:46.535Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d871cbef-1bdd-4002-aefd-aef2aa123aa0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:46.535Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d871cbef-1bdd-4002-aefd-aef2aa123aa0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:35:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:49.578Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d871cbef-1bdd-4002-aefd-aef2aa123aa0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:35:49.579Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d871cbef-1bdd-4002-aefd-aef2aa123aa0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:35:49.579Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d871cbef-1bdd-4002-aefd-aef2aa123aa0 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:35:49.579Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=d871cbef-1bdd-4002-aefd-aef2aa123aa0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:35:49.583Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:49.586Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:49.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04da452b-eba0-48be-a359-cf7b595dcb58 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:49.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04da452b-eba0-48be-a359-cf7b595dcb58 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ca00040 2021-05-04T01:35:49.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04da452b-eba0-48be-a359-cf7b595dcb58 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:49.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04da452b-eba0-48be-a359-cf7b595dcb58 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:35:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:35:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:50.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:35:50.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:35:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:35:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:50.192Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:50.192Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:50.196Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:35:50.196Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:35:50.204Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:35:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:35:52.640Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=04da452b-eba0-48be-a359-cf7b595dcb58 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:35:52.640Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=04da452b-eba0-48be-a359-cf7b595dcb58 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:35:52.643Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:52.645Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:52.645Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d977855-69c7-4bae-bfa8-d7fef838c290 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:52.645Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d977855-69c7-4bae-bfa8-d7fef838c290 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a3e0fc6 2021-05-04T01:35:52.645Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d977855-69c7-4bae-bfa8-d7fef838c290 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:52.645Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d977855-69c7-4bae-bfa8-d7fef838c290 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:35:52.819Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:52.821Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:52.821Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33fb0162-881c-4644-b80a-ebde4dcb3ff2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:52.821Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33fb0162-881c-4644-b80a-ebde4dcb3ff2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c902031 2021-05-04T01:35:52.821Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33fb0162-881c-4644-b80a-ebde4dcb3ff2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:52.821Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33fb0162-881c-4644-b80a-ebde4dcb3ff2 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:35:53.115Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:53.204Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:53.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48897031-eddf-4250-816e-e63a4901be12 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:53.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48897031-eddf-4250-816e-e63a4901be12 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c4e013a 2021-05-04T01:35:53.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48897031-eddf-4250-816e-e63a4901be12 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:53.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48897031-eddf-4250-816e-e63a4901be12 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:35:53.401Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:35:53.407Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=48897031-eddf-4250-816e-e63a4901be12 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:35:53.407Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=48897031-eddf-4250-816e-e63a4901be12 ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:35:53.411Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:53.807Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:53.807Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce9e7e2d-bb9f-4b07-a129-e49483ebce5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:53.808Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce9e7e2d-bb9f-4b07-a129-e49483ebce5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1664614a 2021-05-04T01:35:53.808Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce9e7e2d-bb9f-4b07-a129-e49483ebce5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:53.808Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce9e7e2d-bb9f-4b07-a129-e49483ebce5f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:35:54.016Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ce9e7e2d-bb9f-4b07-a129-e49483ebce5f ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:35:54.016Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ce9e7e2d-bb9f-4b07-a129-e49483ebce5f ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:35:54.020Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:54.403Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:54.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e66fa39-64a5-4003-afac-4de05197b43c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:54.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e66fa39-64a5-4003-afac-4de05197b43c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12ed1206 2021-05-04T01:35:54.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e66fa39-64a5-4003-afac-4de05197b43c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:54.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e66fa39-64a5-4003-afac-4de05197b43c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:35:54.614Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:55.207Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:35:55.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f688021e-a164-48f9-a0d1-ee00354dd8a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:35:55.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f688021e-a164-48f9-a0d1-ee00354dd8a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f5139c2 2021-05-04T01:35:55.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f688021e-a164-48f9-a0d1-ee00354dd8a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:35:55.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f688021e-a164-48f9-a0d1-ee00354dd8a0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:35:55.401Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:00.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:36:03.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:05.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:36:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:36:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:36:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:05.179Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:36:05.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:36:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:05.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:05.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:05.186Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:05.188Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:36:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:15.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:15.626Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:257) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:15.626Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:257) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:15.701Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:15.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb0e09e9-0a82-4418-a849-a771f98c15a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:15.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb0e09e9-0a82-4418-a849-a771f98c15a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@339ab8c3 2021-05-04T01:36:15.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb0e09e9-0a82-4418-a849-a771f98c15a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:15.702Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb0e09e9-0a82-4418-a849-a771f98c15a0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:18.708Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=cb0e09e9-0a82-4418-a849-a771f98c15a0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:36:18.708Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=cb0e09e9-0a82-4418-a849-a771f98c15a0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:36:18.708Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cb0e09e9-0a82-4418-a849-a771f98c15a0 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:36:18.708Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=cb0e09e9-0a82-4418-a849-a771f98c15a0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:36:18.711Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:18.713Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:18.713Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ba1319d-51a2-460e-8fb1-7da1bbb73a15 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:18.713Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ba1319d-51a2-460e-8fb1-7da1bbb73a15 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@641e1cb 2021-05-04T01:36:18.713Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ba1319d-51a2-460e-8fb1-7da1bbb73a15 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:18.713Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9ba1319d-51a2-460e-8fb1-7da1bbb73a15 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:19.210Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9ba1319d-51a2-460e-8fb1-7da1bbb73a15 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:36:19.210Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=9ba1319d-51a2-460e-8fb1-7da1bbb73a15 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:36:19.212Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.214Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c91e3351-5c29-4e45-b76c-cebde0e9b653 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:19.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c91e3351-5c29-4e45-b76c-cebde0e9b653 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a6f1754 2021-05-04T01:36:19.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c91e3351-5c29-4e45-b76c-cebde0e9b653 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:19.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c91e3351-5c29-4e45-b76c-cebde0e9b653 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:19.362Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.364Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7930fe05-9b6d-4ca5-a985-a404800e7224 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:19.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7930fe05-9b6d-4ca5-a985-a404800e7224 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cd5e27a 2021-05-04T01:36:19.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7930fe05-9b6d-4ca5-a985-a404800e7224 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:19.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7930fe05-9b6d-4ca5-a985-a404800e7224 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:19.484Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.486Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.486Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c64c3de3-4015-4e10-b896-775e56b8d9a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:19.486Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c64c3de3-4015-4e10-b896-775e56b8d9a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25130192 2021-05-04T01:36:19.486Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c64c3de3-4015-4e10-b896-775e56b8d9a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:19.486Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c64c3de3-4015-4e10-b896-775e56b8d9a3 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:19.610Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.612Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc4d2b58-c5d7-435c-bc73-f96d0b2681ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:19.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc4d2b58-c5d7-435c-bc73-f96d0b2681ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ced2481 2021-05-04T01:36:19.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc4d2b58-c5d7-435c-bc73-f96d0b2681ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:19.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc4d2b58-c5d7-435c-bc73-f96d0b2681ea PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:19.756Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.758Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e013f095-7bf5-4083-b620-a457de73ffe1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:19.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e013f095-7bf5-4083-b620-a457de73ffe1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e0a4bff 2021-05-04T01:36:19.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e013f095-7bf5-4083-b620-a457de73ffe1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:19.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e013f095-7bf5-4083-b620-a457de73ffe1 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:19.862Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.864Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2664f5-96fa-41be-8008-b19120b39b53 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:19.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2664f5-96fa-41be-8008-b19120b39b53 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5dcfb476 2021-05-04T01:36:19.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2664f5-96fa-41be-8008-b19120b39b53 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:19.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c2664f5-96fa-41be-8008-b19120b39b53 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:19.914Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.916Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:19.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7791d069-1c3b-4b2f-a238-dac3684b95f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:19.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7791d069-1c3b-4b2f-a238-dac3684b95f8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@606d31cc 2021-05-04T01:36:19.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7791d069-1c3b-4b2f-a238-dac3684b95f8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:19.917Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7791d069-1c3b-4b2f-a238-dac3684b95f8 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:20.158Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:36:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:20.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:20.179Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:36:20.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:36:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:20.187Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:36:20.480Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:20.480Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f485dcdd-9c13-446c-90ee-229abfae99e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:20.480Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f485dcdd-9c13-446c-90ee-229abfae99e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65a1c86c 2021-05-04T01:36:20.480Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f485dcdd-9c13-446c-90ee-229abfae99e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:20.480Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f485dcdd-9c13-446c-90ee-229abfae99e9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:36:20.497Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:36:20.586Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f485dcdd-9c13-446c-90ee-229abfae99e9 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:36:20.587Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f485dcdd-9c13-446c-90ee-229abfae99e9 ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:36:20.588Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:20.900Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:20.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fcf5bd3-bc42-4048-a8d4-d4c54c646af5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:20.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fcf5bd3-bc42-4048-a8d4-d4c54c646af5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69f83294 2021-05-04T01:36:20.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fcf5bd3-bc42-4048-a8d4-d4c54c646af5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:20.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fcf5bd3-bc42-4048-a8d4-d4c54c646af5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:36:21.008Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6fcf5bd3-bc42-4048-a8d4-d4c54c646af5 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:36:21.008Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6fcf5bd3-bc42-4048-a8d4-d4c54c646af5 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:36:21.010Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:21.558Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:21.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdcbeeb8-066d-492f-859e-11385592bde5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:21.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdcbeeb8-066d-492f-859e-11385592bde5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49b447d9 2021-05-04T01:36:21.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdcbeeb8-066d-492f-859e-11385592bde5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:21.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdcbeeb8-066d-492f-859e-11385592bde5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:36:21.702Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:30.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:36:33.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:36:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:35.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:36:35.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:36:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:35.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:36:36.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:36:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:36:37.107Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:36:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:36:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:37.110Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:37.114Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:36:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:36:37.115Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:36:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:42.026Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:257) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:42.100Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:42.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f6e8ad6-2030-4d97-801a-1e296e60b380 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:42.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f6e8ad6-2030-4d97-801a-1e296e60b380 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40deec72 2021-05-04T01:36:42.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f6e8ad6-2030-4d97-801a-1e296e60b380 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:42.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f6e8ad6-2030-4d97-801a-1e296e60b380 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:42.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:36:43.525Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2f6e8ad6-2030-4d97-801a-1e296e60b380 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:36:43.525Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=2f6e8ad6-2030-4d97-801a-1e296e60b380 ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:36:43.528Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:43.600Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:43.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f40eab1-b2a4-4275-8d37-ec65d9d17d46 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:43.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f40eab1-b2a4-4275-8d37-ec65d9d17d46 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2bd7aead 2021-05-04T01:36:43.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f40eab1-b2a4-4275-8d37-ec65d9d17d46 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:43.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f40eab1-b2a4-4275-8d37-ec65d9d17d46 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:43.851Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6f40eab1-b2a4-4275-8d37-ec65d9d17d46 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:36:43.851Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=6f40eab1-b2a4-4275-8d37-ec65d9d17d46 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:36:43.900Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:43.902Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:43.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7ec981f-0c1b-43a7-b61a-143586688e01 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:43.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7ec981f-0c1b-43a7-b61a-143586688e01 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39345d1 2021-05-04T01:36:43.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7ec981f-0c1b-43a7-b61a-143586688e01 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:43.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7ec981f-0c1b-43a7-b61a-143586688e01 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:44.108Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.110Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18f2fe78-ec37-4b0b-b47f-ffd5546b59af PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:44.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18f2fe78-ec37-4b0b-b47f-ffd5546b59af PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@687250f5 2021-05-04T01:36:44.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18f2fe78-ec37-4b0b-b47f-ffd5546b59af PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:44.111Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18f2fe78-ec37-4b0b-b47f-ffd5546b59af PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:44.218Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.220Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49c61930-9888-4923-b895-b3fe15f8d062 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:44.220Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49c61930-9888-4923-b895-b3fe15f8d062 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f788cf5 2021-05-04T01:36:44.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49c61930-9888-4923-b895-b3fe15f8d062 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:44.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49c61930-9888-4923-b895-b3fe15f8d062 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:44.401Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.403Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f21954e-d918-415d-bc48-4030d25df90d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:44.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f21954e-d918-415d-bc48-4030d25df90d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49cce371 2021-05-04T01:36:44.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f21954e-d918-415d-bc48-4030d25df90d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:44.404Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f21954e-d918-415d-bc48-4030d25df90d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:44.404Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:36:44.507Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.509Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da53f025-0167-434a-b6c5-d74a694aee59 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:44.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da53f025-0167-434a-b6c5-d74a694aee59 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50689a54 2021-05-04T01:36:44.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da53f025-0167-434a-b6c5-d74a694aee59 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:44.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da53f025-0167-434a-b6c5-d74a694aee59 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:44.615Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=da53f025-0167-434a-b6c5-d74a694aee59 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:36:44.615Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=da53f025-0167-434a-b6c5-d74a694aee59 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:36:44.618Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.621Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56f95411-1a9f-4c1f-b022-b16de8af5692 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:44.622Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56f95411-1a9f-4c1f-b022-b16de8af5692 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ae79d54 2021-05-04T01:36:44.622Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56f95411-1a9f-4c1f-b022-b16de8af5692 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:44.622Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56f95411-1a9f-4c1f-b022-b16de8af5692 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:44.901Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.902Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:44.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71820439-5df7-49af-adab-32f5de6edb82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:44.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71820439-5df7-49af-adab-32f5de6edb82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3163a3d9 2021-05-04T01:36:44.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71820439-5df7-49af-adab-32f5de6edb82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:44.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71820439-5df7-49af-adab-32f5de6edb82 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:45.008Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.010Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a976bb2b-ad31-492d-b063-39b21101e991 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:45.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a976bb2b-ad31-492d-b063-39b21101e991 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f9a09ef 2021-05-04T01:36:45.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a976bb2b-ad31-492d-b063-39b21101e991 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:45.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a976bb2b-ad31-492d-b063-39b21101e991 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:45.133Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.200Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c672bc3-a4de-4d0e-9e6d-2213465d71c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:45.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c672bc3-a4de-4d0e-9e6d-2213465d71c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b349a72 2021-05-04T01:36:45.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c672bc3-a4de-4d0e-9e6d-2213465d71c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:45.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c672bc3-a4de-4d0e-9e6d-2213465d71c1 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:45.401Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.403Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d161186e-0fab-41a9-87d1-cc3a4b24bc92 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:45.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d161186e-0fab-41a9-87d1-cc3a4b24bc92 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d2fbb7d 2021-05-04T01:36:45.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d161186e-0fab-41a9-87d1-cc3a4b24bc92 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:45.403Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d161186e-0fab-41a9-87d1-cc3a4b24bc92 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:45.510Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.511Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=228da3e6-7b61-463b-9b36-9d244ecb4218 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:45.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=228da3e6-7b61-463b-9b36-9d244ecb4218 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31febe6a 2021-05-04T01:36:45.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=228da3e6-7b61-463b-9b36-9d244ecb4218 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:45.511Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=228da3e6-7b61-463b-9b36-9d244ecb4218 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:45.618Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.620Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b766c38-bace-438e-9ca0-df81001e86ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:45.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b766c38-bace-438e-9ca0-df81001e86ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e641ccb 2021-05-04T01:36:45.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b766c38-bace-438e-9ca0-df81001e86ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:45.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b766c38-bace-438e-9ca0-df81001e86ff PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:45.715Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.717Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eee00d6f-8f5b-4dab-a413-d1c13d520e60 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:45.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eee00d6f-8f5b-4dab-a413-d1c13d520e60 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51acfe7f 2021-05-04T01:36:45.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eee00d6f-8f5b-4dab-a413-d1c13d520e60 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:45.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eee00d6f-8f5b-4dab-a413-d1c13d520e60 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:45.807Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.809Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:45.809Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fde869aa-2a47-4136-a76d-71489ee1018a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:45.809Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fde869aa-2a47-4136-a76d-71489ee1018a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31b85364 2021-05-04T01:36:45.809Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fde869aa-2a47-4136-a76d-71489ee1018a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:45.809Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fde869aa-2a47-4136-a76d-71489ee1018a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:46.661Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:46.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:46.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b00d3c-c29f-46cf-a8d7-50a71098b20f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:46.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b00d3c-c29f-46cf-a8d7-50a71098b20f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7571a6d7 2021-05-04T01:36:46.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b00d3c-c29f-46cf-a8d7-50a71098b20f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:46.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b00d3c-c29f-46cf-a8d7-50a71098b20f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:49.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:49.664Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:49.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=558dd45b-ea04-4580-8914-64c993b36345 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:49.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=558dd45b-ea04-4580-8914-64c993b36345 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2bad6ef6 2021-05-04T01:36:49.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=558dd45b-ea04-4580-8914-64c993b36345 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:49.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=558dd45b-ea04-4580-8914-64c993b36345 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:50.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:36:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:50.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:36:50.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:36:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:50.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:36:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:36:50.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:36:52.667Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=558dd45b-ea04-4580-8914-64c993b36345 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:36:52.667Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=558dd45b-ea04-4580-8914-64c993b36345 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:36:52.703Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:52.705Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:52.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a48e5e1-7f01-4a86-b340-47f4dc370741 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:52.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a48e5e1-7f01-4a86-b340-47f4dc370741 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1dc432fa 2021-05-04T01:36:52.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a48e5e1-7f01-4a86-b340-47f4dc370741 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:52.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a48e5e1-7f01-4a86-b340-47f4dc370741 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:55.747Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0a48e5e1-7f01-4a86-b340-47f4dc370741 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:36:55.747Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0a48e5e1-7f01-4a86-b340-47f4dc370741 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:36:55.750Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:55.800Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:55.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c7c9150-b812-4937-9c41-715449152156 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:55.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c7c9150-b812-4937-9c41-715449152156 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@162b354e 2021-05-04T01:36:55.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c7c9150-b812-4937-9c41-715449152156 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:55.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c7c9150-b812-4937-9c41-715449152156 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:36:58.843Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=1c7c9150-b812-4937-9c41-715449152156 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:36:58.843Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=1c7c9150-b812-4937-9c41-715449152156 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:36:58.846Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:58.849Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:36:58.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ee37577-61d1-4777-8065-c09a8f00e9d0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:36:58.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ee37577-61d1-4777-8065-c09a8f00e9d0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ade886d 2021-05-04T01:36:58.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ee37577-61d1-4777-8065-c09a8f00e9d0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:36:58.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ee37577-61d1-4777-8065-c09a8f00e9d0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:37:01.908Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7ee37577-61d1-4777-8065-c09a8f00e9d0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:37:01.909Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7ee37577-61d1-4777-8065-c09a8f00e9d0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:37:01.912Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:01.914Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:01.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c09f437-90ca-4fcf-9824-8693a4b41334 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:01.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c09f437-90ca-4fcf-9824-8693a4b41334 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68c30250 2021-05-04T01:37:01.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c09f437-90ca-4fcf-9824-8693a4b41334 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:01.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c09f437-90ca-4fcf-9824-8693a4b41334 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:37:03.303Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:37:03.303Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:37:04.303Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:37:04.950Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7c09f437-90ca-4fcf-9824-8693a4b41334 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:37:04.950Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7c09f437-90ca-4fcf-9824-8693a4b41334 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:37:05.002Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:05.004Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:05.004Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f15fe146-360f-4a8a-a910-989fec3e6759 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:05.004Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f15fe146-360f-4a8a-a910-989fec3e6759 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29b3e3bb 2021-05-04T01:37:05.004Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f15fe146-360f-4a8a-a910-989fec3e6759 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:05.004Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f15fe146-360f-4a8a-a910-989fec3e6759 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:37:05.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:37:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:05.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:05.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:05.186Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:37:05.187Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:37:05.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:05.189Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:05.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:05.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:05.192Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:05.192Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:05.193Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:37:05.304Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:37:06.304Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:37:07.304Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:37:08.059Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f15fe146-360f-4a8a-a910-989fec3e6759 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:37:08.059Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f15fe146-360f-4a8a-a910-989fec3e6759 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:37:08.060Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f15fe146-360f-4a8a-a910-989fec3e6759 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:37:08.060Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f15fe146-360f-4a8a-a910-989fec3e6759 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:37:08.063Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:08.066Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:08.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1d672cb-2c70-4957-92d1-8217e961310f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:08.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1d672cb-2c70-4957-92d1-8217e961310f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1043b36b 2021-05-04T01:37:08.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1d672cb-2c70-4957-92d1-8217e961310f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:08.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1d672cb-2c70-4957-92d1-8217e961310f PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:37:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:37:11.116Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e1d672cb-2c70-4957-92d1-8217e961310f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:37:11.116Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e1d672cb-2c70-4957-92d1-8217e961310f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:37:11.120Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:11.122Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:11.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3fcc479-a690-413f-9a1b-45cab1028433 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:11.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3fcc479-a690-413f-9a1b-45cab1028433 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@159e5e5b 2021-05-04T01:37:11.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3fcc479-a690-413f-9a1b-45cab1028433 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:11.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3fcc479-a690-413f-9a1b-45cab1028433 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:37:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:37:13.701Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:13.703Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c46d3b84-ab17-4920-a037-95f477542e98 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c46d3b84-ab17-4920-a037-95f477542e98 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f3ba0d4 2021-05-04T01:37:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c46d3b84-ab17-4920-a037-95f477542e98 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:13.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c46d3b84-ab17-4920-a037-95f477542e98 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:37:14.301Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:37:16.705Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:16.810Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:16.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456ff64d-6178-4cb2-a9eb-441aa9f7beaa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:16.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456ff64d-6178-4cb2-a9eb-441aa9f7beaa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a55d19f 2021-05-04T01:37:16.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456ff64d-6178-4cb2-a9eb-441aa9f7beaa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:16.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456ff64d-6178-4cb2-a9eb-441aa9f7beaa PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:16.922Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=456ff64d-6178-4cb2-a9eb-441aa9f7beaa ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:37:16.922Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=456ff64d-6178-4cb2-a9eb-441aa9f7beaa ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:37:19.662Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:20.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:37:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:20.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:20.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:37:20.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:37:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:37:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:20.185Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:37:20.845Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:20.845Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4699a87-82a4-4d05-8ece-6bb99c24d2be PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:20.845Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4699a87-82a4-4d05-8ece-6bb99c24d2be PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1dde2408 2021-05-04T01:37:20.845Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4699a87-82a4-4d05-8ece-6bb99c24d2be PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:20.845Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4699a87-82a4-4d05-8ece-6bb99c24d2be PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:22.701Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:37:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:35.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:35.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:37:35.185Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:37:35.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:35.187Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:35.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:35.188Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:35.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:35.190Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:35.191Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:37:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:37:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:37:37.109Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:37:37.109Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:37:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:37.115Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:37:37.116Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:37:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:37:38.852Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:38.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=105c578c-64b5-463d-b945-512b84378470 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:38.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=105c578c-64b5-463d-b945-512b84378470 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bde767d 2021-05-04T01:37:38.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=105c578c-64b5-463d-b945-512b84378470 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:38.853Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=105c578c-64b5-463d-b945-512b84378470 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:38.958Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:39.675Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:39.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e317ce4f-4a79-483c-a433-ad362f5cbdc7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:39.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e317ce4f-4a79-483c-a433-ad362f5cbdc7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39d15ad2 2021-05-04T01:37:39.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e317ce4f-4a79-483c-a433-ad362f5cbdc7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:39.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e317ce4f-4a79-483c-a433-ad362f5cbdc7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:39.808Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:40.255Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:40.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d02fe837-37a7-4de3-ad2b-4ff1ab783ef8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:40.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d02fe837-37a7-4de3-ad2b-4ff1ab783ef8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d5cbd5e 2021-05-04T01:37:40.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d02fe837-37a7-4de3-ad2b-4ff1ab783ef8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:40.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d02fe837-37a7-4de3-ad2b-4ff1ab783ef8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:40.460Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:40.800Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:40.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7706f43b-5430-479b-89e1-57e3f7db0ab0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:40.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7706f43b-5430-479b-89e1-57e3f7db0ab0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14e3a99 2021-05-04T01:37:40.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7706f43b-5430-479b-89e1-57e3f7db0ab0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:40.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7706f43b-5430-479b-89e1-57e3f7db0ab0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:40.924Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:41.180Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:41.180Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8b92621-9fac-40d5-9738-39acbd2d50ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:41.181Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8b92621-9fac-40d5-9738-39acbd2d50ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@795bc953 2021-05-04T01:37:41.181Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8b92621-9fac-40d5-9738-39acbd2d50ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:41.181Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8b92621-9fac-40d5-9738-39acbd2d50ca PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:41.220Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:41.500Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:41.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06cb092e-8c28-4fed-a420-b620b2f85449 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:41.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06cb092e-8c28-4fed-a420-b620b2f85449 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a6f0b60 2021-05-04T01:37:41.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06cb092e-8c28-4fed-a420-b620b2f85449 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:41.500Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06cb092e-8c28-4fed-a420-b620b2f85449 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:41.632Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:41.800Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:41.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=908a1a5d-ba54-4d62-8bfc-5e7c00f2eef4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:41.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=908a1a5d-ba54-4d62-8bfc-5e7c00f2eef4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42d1f93e 2021-05-04T01:37:41.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=908a1a5d-ba54-4d62-8bfc-5e7c00f2eef4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:41.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=908a1a5d-ba54-4d62-8bfc-5e7c00f2eef4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:43.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:44.500Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a601c8c-dd17-4e32-a39a-844c046ebcc4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:37:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a601c8c-dd17-4e32-a39a-844c046ebcc4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@323b990f 2021-05-04T01:37:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a601c8c-dd17-4e32-a39a-844c046ebcc4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:37:44.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a601c8c-dd17-4e32-a39a-844c046ebcc4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:37:46.663Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:37:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:37:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:50.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:50.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:37:50.177Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:37:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:50.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:37:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:37:50.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:38:01.700Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:01.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d09965c5-dfd6-4fb7-ae13-71ca29ac3e18 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:01.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d09965c5-dfd6-4fb7-ae13-71ca29ac3e18 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@158ddf4a 2021-05-04T01:38:01.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d09965c5-dfd6-4fb7-ae13-71ca29ac3e18 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:01.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d09965c5-dfd6-4fb7-ae13-71ca29ac3e18 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:38:01.899Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:02.310Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:02.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8079290a-2f57-495a-9987-16af85c25930 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:02.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8079290a-2f57-495a-9987-16af85c25930 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@278bf290 2021-05-04T01:38:02.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8079290a-2f57-495a-9987-16af85c25930 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:02.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8079290a-2f57-495a-9987-16af85c25930 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:38:02.500Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:02.900Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:02.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c51a779-76a8-43a7-b585-08469517d005 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:02.900Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c51a779-76a8-43a7-b585-08469517d005 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a5ef756 2021-05-04T01:38:02.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c51a779-76a8-43a7-b585-08469517d005 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:02.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c51a779-76a8-43a7-b585-08469517d005 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:38:03.100Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:04.501Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:04.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f274de82-337d-47e6-8ee5-539a5c4a32a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:04.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f274de82-337d-47e6-8ee5-539a5c4a32a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@443e8f3 2021-05-04T01:38:04.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f274de82-337d-47e6-8ee5-539a5c4a32a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:04.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f274de82-337d-47e6-8ee5-539a5c4a32a0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:38:04.622Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:38:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:05.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:05.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:05.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:38:05.175Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:38:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:05.177Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:05.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:05.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:05.183Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:38:20.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:38:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:20.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:38:20.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:20.176Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:38:20.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:38:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:20.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:38:20.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:20.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:38:24.727Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:24.729Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:24.729Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55487bde-8a3e-4da5-9ce9-9ba868386d7c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:24.729Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55487bde-8a3e-4da5-9ce9-9ba868386d7c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fcf00be 2021-05-04T01:38:24.729Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55487bde-8a3e-4da5-9ce9-9ba868386d7c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:24.729Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55487bde-8a3e-4da5-9ce9-9ba868386d7c PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:25.057Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=55487bde-8a3e-4da5-9ce9-9ba868386d7c ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:38:25.057Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=55487bde-8a3e-4da5-9ce9-9ba868386d7c ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:38:25.101Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:25.105Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:25.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b1008c0-ddcf-45c6-b545-674e92b924f0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:25.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b1008c0-ddcf-45c6-b545-674e92b924f0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72862a5 2021-05-04T01:38:25.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b1008c0-ddcf-45c6-b545-674e92b924f0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:25.105Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b1008c0-ddcf-45c6-b545-674e92b924f0 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:25.209Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:25.211Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:25.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f15c0002-c922-41d9-a221-351f5b82137d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:25.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f15c0002-c922-41d9-a221-351f5b82137d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a51d151 2021-05-04T01:38:25.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f15c0002-c922-41d9-a221-351f5b82137d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:25.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f15c0002-c922-41d9-a221-351f5b82137d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:25.322Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:25.324Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:25.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=316e3fe2-936e-48ef-8b5f-4ad00e0ba8a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:25.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=316e3fe2-936e-48ef-8b5f-4ad00e0ba8a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1cea698 2021-05-04T01:38:25.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=316e3fe2-936e-48ef-8b5f-4ad00e0ba8a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:25.324Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=316e3fe2-936e-48ef-8b5f-4ad00e0ba8a6 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:25.565Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:25.600Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:25.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d330117-9063-4a1a-b822-5401233847fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:25.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d330117-9063-4a1a-b822-5401233847fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6038cef 2021-05-04T01:38:25.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d330117-9063-4a1a-b822-5401233847fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:25.601Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d330117-9063-4a1a-b822-5401233847fd PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:26.378Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.381Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.381Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1d77f31-933a-4bbb-bf39-03107ffa1fce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:26.381Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1d77f31-933a-4bbb-bf39-03107ffa1fce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42faebc0 2021-05-04T01:38:26.381Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1d77f31-933a-4bbb-bf39-03107ffa1fce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:26.381Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1d77f31-933a-4bbb-bf39-03107ffa1fce PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:26.466Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.468Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9061bb96-fa88-45ed-bba7-1bd8ac06396d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:26.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9061bb96-fa88-45ed-bba7-1bd8ac06396d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56b56197 2021-05-04T01:38:26.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9061bb96-fa88-45ed-bba7-1bd8ac06396d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:26.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9061bb96-fa88-45ed-bba7-1bd8ac06396d PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:26.563Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.565Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01e788be-7577-4f22-85ca-a7ef007c00cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:26.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01e788be-7577-4f22-85ca-a7ef007c00cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22bbe7bf 2021-05-04T01:38:26.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01e788be-7577-4f22-85ca-a7ef007c00cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:26.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01e788be-7577-4f22-85ca-a7ef007c00cd PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:26.674Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.676Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2611d099-231e-4460-ac19-f07300b4c40c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:26.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2611d099-231e-4460-ac19-f07300b4c40c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@384573f3 2021-05-04T01:38:26.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2611d099-231e-4460-ac19-f07300b4c40c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:26.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2611d099-231e-4460-ac19-f07300b4c40c PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:26.767Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.769Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:26.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d036d17-6e47-441e-888d-4f005cb76d31 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:26.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d036d17-6e47-441e-888d-4f005cb76d31 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d742bd0 2021-05-04T01:38:26.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d036d17-6e47-441e-888d-4f005cb76d31 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:26.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d036d17-6e47-441e-888d-4f005cb76d31 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:38:28.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:29.867Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:29.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70b746ae-49a4-4620-bb69-e9c540d270e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:29.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70b746ae-49a4-4620-bb69-e9c540d270e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50b04e30 2021-05-04T01:38:29.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70b746ae-49a4-4620-bb69-e9c540d270e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:29.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70b746ae-49a4-4620-bb69-e9c540d270e1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:38:29.907Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=70b746ae-49a4-4620-bb69-e9c540d270e1 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:38:29.907Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=70b746ae-49a4-4620-bb69-e9c540d270e1 ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:38:31.663Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:31.991Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:31.991Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf73b5a5-532a-44cd-a915-7d597b9eba9d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:31.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf73b5a5-532a-44cd-a915-7d597b9eba9d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2dcbdd86 2021-05-04T01:38:31.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf73b5a5-532a-44cd-a915-7d597b9eba9d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:31.992Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf73b5a5-532a-44cd-a915-7d597b9eba9d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:38:31.994Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=bf73b5a5-532a-44cd-a915-7d597b9eba9d PartnerName=SDC.BE I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://sdc-onboarding-be.onap:8445: The target server failed to respond 2021-05-04T01:38:31.994Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=bf73b5a5-532a-44cd-a915-7d597b9eba9d PartnerName=SDC.BE Retrying request to {s}->https://sdc-onboarding-be.onap:8445 2021-05-04T01:38:31.998Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bf73b5a5-532a-44cd-a915-7d597b9eba9d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:38:31.999Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bf73b5a5-532a-44cd-a915-7d597b9eba9d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:38:33.203Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:38:33.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:38:34.700Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:35.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:38:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:35.171Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:35.173Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:35.175Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:38:35.176Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:38:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:35.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:35.179Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:35.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:35.182Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:38:36.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:38:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:38:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:38:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:38:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:38:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:37.113Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:37.115Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:37.116Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:37.116Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:37.118Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:38:37.118Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:38:37.119Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:38:37.261Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:37.261Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:38:37.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a92fd63c-3180-4715-9f48-9f6fb9485516 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:37.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a92fd63c-3180-4715-9f48-9f6fb9485516 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32247f8c 2021-05-04T01:38:37.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a92fd63c-3180-4715-9f48-9f6fb9485516 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:37.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a92fd63c-3180-4715-9f48-9f6fb9485516 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:38:37.263Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a92fd63c-3180-4715-9f48-9f6fb9485516 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:38:37.264Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a92fd63c-3180-4715-9f48-9f6fb9485516 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:38:37.264Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a92fd63c-3180-4715-9f48-9f6fb9485516 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:38:37.264Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=a92fd63c-3180-4715-9f48-9f6fb9485516 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:38:37.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:38.521Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:38.521Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=391bd62c-db38-455a-8603-2ef03720cb69 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:38.521Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=391bd62c-db38-455a-8603-2ef03720cb69 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13e3a05b 2021-05-04T01:38:38.521Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=391bd62c-db38-455a-8603-2ef03720cb69 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:38.521Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=391bd62c-db38-455a-8603-2ef03720cb69 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:38:38.524Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=391bd62c-db38-455a-8603-2ef03720cb69 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:38:38.524Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=391bd62c-db38-455a-8603-2ef03720cb69 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:38:39.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:38:40.662Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:41.400Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:41.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed3cbd42-5efd-4c31-ad52-03dfeb18fb07 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:38:41.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed3cbd42-5efd-4c31-ad52-03dfeb18fb07 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2bc8e802 2021-05-04T01:38:41.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed3cbd42-5efd-4c31-ad52-03dfeb18fb07 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:38:41.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed3cbd42-5efd-4c31-ad52-03dfeb18fb07 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:38:42.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:38:42.488Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:38:43.868Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:832) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1988) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:212) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:38:44.419Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=ed3cbd42-5efd-4c31-ad52-03dfeb18fb07 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:38:44.419Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=ed3cbd42-5efd-4c31-ad52-03dfeb18fb07 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:38:44.423Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:38:45.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:38:48.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:38:49.955Z [ApplicationDataTypeCacheThread-1] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=FetchDataTypes Internal Connection problem. Description: Failed to fetch data types from graph(cache) 2021-05-04T01:38:49.958Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:38:34.503122Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:38:50.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:38:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:50.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:50.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:50.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:50.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:50.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:38:50.179Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:38:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:50.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:50.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:38:50.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:38:50.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:38:51.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:38:54.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:38:57.202Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:00.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:03.206Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:05.169Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:39:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:05.172Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:05.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:05.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:05.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:05.177Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:39:05.178Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:39:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:05.180Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:05.181Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:05.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:05.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:05.184Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:39:06.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:09.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:10.401Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:39:10.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c111eec4-90a5-4d73-ae1c-a398bc9eaa6b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:10.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c111eec4-90a5-4d73-ae1c-a398bc9eaa6b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e7ababa 2021-05-04T01:39:10.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c111eec4-90a5-4d73-ae1c-a398bc9eaa6b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:10.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c111eec4-90a5-4d73-ae1c-a398bc9eaa6b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:11.028Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:38:54.458301Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:39:12.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:15.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:15.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ef15f37-5cbd-448c-8d01-f2536779b678 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:15.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ef15f37-5cbd-448c-8d01-f2536779b678 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@216aabcf 2021-05-04T01:39:15.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ef15f37-5cbd-448c-8d01-f2536779b678 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:15.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ef15f37-5cbd-448c-8d01-f2536779b678 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:15.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d4c20db-9a79-42f7-bfa9-c99c24679d72 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:15.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d4c20db-9a79-42f7-bfa9-c99c24679d72 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78caa39c 2021-05-04T01:39:15.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d4c20db-9a79-42f7-bfa9-c99c24679d72 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:15.901Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d4c20db-9a79-42f7-bfa9-c99c24679d72 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:16.051Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:84) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:80) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:39:16.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f42fe7d0-a711-4417-9dc1-ed0870cb0ef3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:16.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f42fe7d0-a711-4417-9dc1-ed0870cb0ef3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53ce7955 2021-05-04T01:39:16.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f42fe7d0-a711-4417-9dc1-ed0870cb0ef3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:16.700Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f42fe7d0-a711-4417-9dc1-ed0870cb0ef3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:17.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2d61328-44d6-4663-bfab-9bd19c057e69 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:17.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2d61328-44d6-4663-bfab-9bd19c057e69 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68ef0b4a 2021-05-04T01:39:17.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2d61328-44d6-4663-bfab-9bd19c057e69 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:17.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2d61328-44d6-4663-bfab-9bd19c057e69 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:18.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:18.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e109c466-5e00-4919-a760-724f0e7322f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:18.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e109c466-5e00-4919-a760-724f0e7322f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c2ff14b 2021-05-04T01:39:18.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e109c466-5e00-4919-a760-724f0e7322f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:18.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e109c466-5e00-4919-a760-724f0e7322f6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:18.628Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7fa616e9-cec5-416a-95e6-858ce3d45007 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:18.628Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7fa616e9-cec5-416a-95e6-858ce3d45007 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31546711 2021-05-04T01:39:18.628Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7fa616e9-cec5-416a-95e6-858ce3d45007 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:18.628Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7fa616e9-cec5-416a-95e6-858ce3d45007 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:18.980Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bfd4d0f-31c2-491c-bf09-9e606b32eddf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:18.980Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bfd4d0f-31c2-491c-bf09-9e606b32eddf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1170a4bf 2021-05-04T01:39:18.980Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bfd4d0f-31c2-491c-bf09-9e606b32eddf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:18.980Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bfd4d0f-31c2-491c-bf09-9e606b32eddf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:19.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad32677b-89e1-4b21-9a9f-ff20a4ec5b5c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:19.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad32677b-89e1-4b21-9a9f-ff20a4ec5b5c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1555880 2021-05-04T01:39:19.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad32677b-89e1-4b21-9a9f-ff20a4ec5b5c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:19.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad32677b-89e1-4b21-9a9f-ff20a4ec5b5c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:19.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f04d6d72-6ed4-4d7c-82c7-9be30917e368 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:19.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f04d6d72-6ed4-4d7c-82c7-9be30917e368 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62935448 2021-05-04T01:39:19.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f04d6d72-6ed4-4d7c-82c7-9be30917e368 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:19.552Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f04d6d72-6ed4-4d7c-82c7-9be30917e368 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:19.921Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab952ba9-30ea-4d91-93c3-9a9a08e9e31a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:19.921Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab952ba9-30ea-4d91-93c3-9a9a08e9e31a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70eddbe4 2021-05-04T01:39:19.921Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab952ba9-30ea-4d91-93c3-9a9a08e9e31a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:19.921Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab952ba9-30ea-4d91-93c3-9a9a08e9e31a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:20.170Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:39:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:20.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:39:20.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:20.178Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:20.179Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:39:20.181Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:39:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:20.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:20.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:20.184Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:20.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:20.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:20.186Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:39:20.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6559780c-db1d-42d6-814b-7ed7e6f8f78a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:20.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6559780c-db1d-42d6-814b-7ed7e6f8f78a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64acdc1a 2021-05-04T01:39:20.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6559780c-db1d-42d6-814b-7ed7e6f8f78a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:20.250Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6559780c-db1d-42d6-814b-7ed7e6f8f78a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:20.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01d0ba5b-1f25-4130-bed6-4a2233094ab6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:20.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01d0ba5b-1f25-4130-bed6-4a2233094ab6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f961957 2021-05-04T01:39:20.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01d0ba5b-1f25-4130-bed6-4a2233094ab6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:20.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01d0ba5b-1f25-4130-bed6-4a2233094ab6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:21.053Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:38:54.458301Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:39:21.201Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:23.215Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:39:24.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a6c69c7-f82b-4d13-872b-7886893bdc4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:24.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a6c69c7-f82b-4d13-872b-7886893bdc4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@375f25f9 2021-05-04T01:39:24.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a6c69c7-f82b-4d13-872b-7886893bdc4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:24.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a6c69c7-f82b-4d13-872b-7886893bdc4a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c9c8f77-2b66-48e5-b73f-e87785eb3427 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c9c8f77-2b66-48e5-b73f-e87785eb3427 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2973ea0d 2021-05-04T01:39:26.045Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c9c8f77-2b66-48e5-b73f-e87785eb3427 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:26.046Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c9c8f77-2b66-48e5-b73f-e87785eb3427 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:26.051Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1c9c8f77-2b66-48e5-b73f-e87785eb3427 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:39:26.051Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1c9c8f77-2b66-48e5-b73f-e87785eb3427 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:39:29.349Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb119589-34f6-4023-9581-59e932d91bae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:29.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb119589-34f6-4023-9581-59e932d91bae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b3daf35 2021-05-04T01:39:29.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb119589-34f6-4023-9581-59e932d91bae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:29.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb119589-34f6-4023-9581-59e932d91bae PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:32.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2db8801e-cff5-4e58-8b75-89b6a59f0557 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:32.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2db8801e-cff5-4e58-8b75-89b6a59f0557 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@541e57a4 2021-05-04T01:39:32.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2db8801e-cff5-4e58-8b75-89b6a59f0557 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:32.547Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2db8801e-cff5-4e58-8b75-89b6a59f0557 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:35.171Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:39:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:35.174Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:35.175Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:35.176Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:35.178Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:39:35.180Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-05-04T01:39:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:35.182Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:35.183Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:35.185Z [UEB-Health-Check-Thread] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:35.186Z [UEB-Health-Check-Thread] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:39:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=901731f1-ff16-4aea-adf6-29f0dab0c309 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=901731f1-ff16-4aea-adf6-29f0dab0c309 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1aa0c033 2021-05-04T01:39:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=901731f1-ff16-4aea-adf6-29f0dab0c309 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:35.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=901731f1-ff16-4aea-adf6-29f0dab0c309 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:37.107Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-05-04T01:39:37.107Z [TopicPollingThread-1] WARN c.a.nsa.apiClient.http.HostSelector RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling All hosts were blacklisted; reverting to full set of hosts. 2021-05-04T01:39:37.108Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-05-04T01:39:37.108Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-05-04T01:39:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:37.111Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:37.112Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:39:37.114Z [TopicPollingThread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:39:37.117Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:39:37.117Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=200 TargetEntity=fetchFromTopic Connection problem towards U-EB server. Reason: 500 2021-05-04T01:39:37.118Z [TopicPollingThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=fetchFromTopic TargetServiceName=null ErrorCode=500 TargetEntity=distributionEngineStatusPolling Error occured during access to U-EB Server. Operation: failed to fetch messages from topic SDC-DISTR-STATUS-TOPIC-AUTO error: CambriaErrorResponse [operationStatus=CONNNECTION_ERROR, httpCode=500, variables=[]] 2021-05-04T01:39:37.592Z [JanusGraph-Health-Check-Task] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.isGraphOpen(JanusGraphClient.java:346) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckScheduledTask.run(JanusGraphClient.java:95) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:39:38.093Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c48f040c-885e-4ebb-a3c2-44a9c8f38028 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:39:38.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c48f040c-885e-4ebb-a3c2-44a9c8f38028 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c3572e8 2021-05-04T01:39:38.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c48f040c-885e-4ebb-a3c2-44a9c8f38028 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:39:38.094Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c48f040c-885e-4ebb-a3c2-44a9c8f38028 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:39:38.698Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:39:38.698Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:40:07.292Z [main] WARN o.o.s.c.util.YamlToObjectConverter RequestID=null ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= The file /var/lib/jetty/config/catalog-be/neo4j-errors-configuration.yaml cannot be found. Ignore reading configuration. 2021-05-04T01:40:07.451Z [main] INFO o.o.s.be.listen.BEAppContextListener RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ASDC version from MANIFEST is 1.8.5 2021-05-04T01:40:07.498Z [main] INFO o.o.s.b.m.BeMonitoringService RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= BE monitoring service is disabled 2021-05-04T01:40:07.499Z [main] INFO o.s.web.context.ContextLoader RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Root WebApplicationContext: initialization started 2021-05-04T01:40:09.606Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphClient created 2021-05-04T01:40:09.615Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** createGraph started ** 2021-05-04T01:40:09.615Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T01:40:09.615Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T01:40:09.909Z [main] INFO com.datastax.driver.core RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= DataStax Java driver 3.8.0 for Apache Cassandra 2021-05-04T01:40:09.914Z [main] INFO c.d.driver.core.GuavaCompatibility RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Detected Guava >= 19 in the classpath, using modern compatibility layer 2021-05-04T01:40:10.507Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T01:40:10.725Z [main] INFO com.datastax.driver.core.NettyUtil RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Found Netty's native epoll transport in the classpath, using it 2021-05-04T01:40:14.676Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:40:14.678Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:40:14.678Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.95.7:9042 added 2021-05-04T01:40:14.678Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:40:17.559Z [main] WARN o.j.g.c.GraphDatabaseConfiguration RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Local setting storage.cql.replication-strategy-options=[Pod, 3] (Type: FIXED) is overridden by globally managed value ([Pod, 3]). Use the ManagementSystem interface instead of the local configuration to control this setting. 2021-05-04T01:40:19.683Z [main] INFO o.j.g.c.GraphDatabaseConfiguration RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Generated unique-instance-id=0af2eb9a251-dev-sdc-be-69f59769db-d8vzh1 2021-05-04T01:40:19.705Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T01:40:21.380Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:40:21.380Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:40:21.381Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.95.7:9042 added 2021-05-04T01:40:21.381Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:40:22.489Z [main] INFO org.janusgraph.diskstorage.Backend RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Initiated backend operations thread pool of size 2 2021-05-04T01:40:25.651Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T01:40:25.654Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Scheduling reconnect task org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask@60a4595e with interval of 3 seconds 2021-05-04T01:40:25.655Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T01:40:25.656Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T01:40:25.660Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T01:40:25.704Z [main] INFO o.o.s.b.d.j.JanusGraphGenericDao RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphGenericDao created 2021-05-04T01:40:26.002Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= All sdc keyspaces are : [sdcartifact, sdcrepository, sdcaudit] 2021-05-04T01:40:26.003Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connecting to node: [cassandra.onap] port: 9042. 2021-05-04T01:40:26.004Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connection timeout in seconds : 120 2021-05-04T01:40:26.004Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T01:40:26.004Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T01:40:26.004Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T01:40:26.004Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T01:40:26.004Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** cluster created 2021-05-04T01:40:26.004Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= creating cluster for Cassandra Health Check. 2021-05-04T01:40:27.643Z [JanusGraph-Reconnect-Task] INFO c.d.d.c.p.DCAwareRoundRobinPolicy Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:40:27.643Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:40:27.643Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.95.7:9042 added 2021-05-04T01:40:27.643Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:40:28.151Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:40:28.151Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:40:28.152Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.95.7:9042 added 2021-05-04T01:40:28.152Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.235.132:9042 added 2021-05-04T01:40:28.152Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Cluster Metadata: com.datastax.driver.core.Metadata@3df496d9 2021-05-04T01:40:28.152Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= keyspace : sdcartifact , replication: {Pod=3, class=org.apache.cassandra.locator.NetworkTopologyStrategy} 2021-05-04T01:40:28.152Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= keyspace : sdcrepository , replication: {Pod=3, class=org.apache.cassandra.locator.NetworkTopologyStrategy} 2021-05-04T01:40:28.152Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= keyspace : sdcaudit , replication: {Pod=3, class=org.apache.cassandra.locator.NetworkTopologyStrategy} 2021-05-04T01:40:28.152Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= maxReplication Factor is: 3 2021-05-04T01:40:28.152Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localQuorum is: 2 2021-05-04T01:40:28.152Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Health Check formula : Replication Factor – Local_Quorum = 1 2021-05-04T01:40:28.178Z [main] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ECOMP Portal health check configuration is missing. 2021-05-04T01:40:28.197Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Enter init method of SwitchoverDetector 2021-05-04T01:40:28.198Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null switchover detector service is disabled 2021-05-04T01:40:28.899Z [JanusGraph-Reconnect-Task] WARN o.j.g.c.GraphDatabaseConfiguration Local setting storage.cql.replication-strategy-options=[Pod, 3] (Type: FIXED) is overridden by globally managed value ([Pod, 3]). Use the ManagementSystem interface instead of the local configuration to control this setting. 2021-05-04T01:40:28.902Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null creating cluster for Cassandra for monitoring. 2021-05-04T01:40:28.902Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra session is com.datastax.driver.core.SessionManager@1a44ad96 2021-05-04T01:40:28.902Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The number of cassandra nodes is:3 2021-05-04T01:40:28.904Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra down nodes number is 0 2021-05-04T01:40:29.734Z [main] INFO o.o.s.be.dao.jsongraph.JanusGraphDao RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** JanusGraphDao created 2021-05-04T01:40:30.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa00b67d-7270-4713-8c7c-198740f5ddf5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:30.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa00b67d-7270-4713-8c7c-198740f5ddf5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a14fd48 2021-05-04T01:40:30.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa00b67d-7270-4713-8c7c-198740f5ddf5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:30.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa00b67d-7270-4713-8c7c-198740f5ddf5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:40:31.008Z [JanusGraph-Reconnect-Task] INFO o.j.g.c.GraphDatabaseConfiguration Generated unique-instance-id=0af2eb9a251-dev-sdc-be-69f59769db-d8vzh2 2021-05-04T01:40:31.011Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T01:40:31.649Z [JanusGraph-Reconnect-Task] INFO c.d.d.c.p.DCAwareRoundRobinPolicy Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:40:31.649Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:40:31.650Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.95.7:9042 added 2021-05-04T01:40:31.650Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:40:32.424Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T01:40:32.425Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T01:40:32.426Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T01:40:32.426Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using native clock to generate timestamps. 2021-05-04T01:40:32.426Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient created 2021-05-04T01:40:32.548Z [JanusGraph-Reconnect-Task] INFO org.janusgraph.diskstorage.Backend Initiated backend operations thread pool of size 2 2021-05-04T01:40:32.783Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:40:32.784Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:40:32.784Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.95.7:9042 added 2021-05-04T01:40:32.784Z [main] INFO com.datastax.driver.core.Cluster RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.235.132:9042 added 2021-05-04T01:40:33.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fa5c2a7-7c46-4a59-a5a7-cf9a27ab0942 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:33.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fa5c2a7-7c46-4a59-a5a7-cf9a27ab0942 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74e1dcb6 2021-05-04T01:40:33.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fa5c2a7-7c46-4a59-a5a7-cf9a27ab0942 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:33.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fa5c2a7-7c46-4a59-a5a7-cf9a27ab0942 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:40:33.907Z [main] INFO o.o.s.b.d.c.AuditCassandraDao RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** AuditCassandraDao created 2021-05-04T01:40:34.061Z [main] INFO o.o.s.b.c.d.e.DmaapClientFactory RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1620092387, consumerId=sdc-AUTO1-1620092387, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/var/lib/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-05-04T01:40:34.131Z [main] ERROR o.o.s.be.catalog.impl.DmaapProducer RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-05-04T01:40:34.583Z [JanusGraph-Reconnect-Task] INFO o.j.diskstorage.log.kcvs.KCVSLog Loaded unidentified ReadMarker start time 2021-05-04T01:40:34.583088Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@39120d80 2021-05-04T01:40:34.812Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** JanusGraph graph created 2021-05-04T01:40:34.812Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** Cancelling JanusGraph reconnect task 2021-05-04T01:40:36.402Z [main] INFO o.o.s.b.d.c.ArtifactCassandraDao RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** ArtifactCassandraDao created 2021-05-04T01:40:36.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a9691b1-29cd-408a-9e62-46614d238794 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:36.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a9691b1-29cd-408a-9e62-46614d238794 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10d8c944 2021-05-04T01:40:36.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a9691b1-29cd-408a-9e62-46614d238794 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:36.525Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a9691b1-29cd-408a-9e62-46614d238794 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:40:39.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3d235b5-43be-413a-bd34-09bfdfe17574 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:39.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3d235b5-43be-413a-bd34-09bfdfe17574 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25c1e70e 2021-05-04T01:40:39.027Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3d235b5-43be-413a-bd34-09bfdfe17574 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:39.027Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3d235b5-43be-413a-bd34-09bfdfe17574 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:40:42.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4973858e-7e29-4ebc-b43e-dfd0344ee3d2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:42.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4973858e-7e29-4ebc-b43e-dfd0344ee3d2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27e64da4 2021-05-04T01:40:42.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4973858e-7e29-4ebc-b43e-dfd0344ee3d2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:42.504Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4973858e-7e29-4ebc-b43e-dfd0344ee3d2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:40:45.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e7466a4-67cd-428b-a0be-8ec728ab6961 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:45.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e7466a4-67cd-428b-a0be-8ec728ab6961 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d182182 2021-05-04T01:40:45.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e7466a4-67cd-428b-a0be-8ec728ab6961 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:45.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e7466a4-67cd-428b-a0be-8ec728ab6961 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:40:48.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64ca0a2a-c97b-4dcd-be02-d15d8ec3e9c7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:48.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64ca0a2a-c97b-4dcd-be02-d15d8ec3e9c7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bdc418c 2021-05-04T01:40:48.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64ca0a2a-c97b-4dcd-be02-d15d8ec3e9c7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:48.205Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64ca0a2a-c97b-4dcd-be02-d15d8ec3e9c7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:40:51.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1bfd607-3cb9-4974-a8db-fdeb08a024c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:51.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1bfd607-3cb9-4974-a8db-fdeb08a024c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@218f3e6e 2021-05-04T01:40:51.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1bfd607-3cb9-4974-a8db-fdeb08a024c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:51.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1bfd607-3cb9-4974-a8db-fdeb08a024c9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:40:56.278Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getChildrenNodes(JanusGraphGenericDao.java:1336) at org.openecomp.sdc.be.dao.janusgraph.HealingJanusGraphGenericDao.getChildrenNodes(HealingJanusGraphGenericDao.java:57) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getChildrenNodes(JanusGraphGenericDao.java:1390) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.findPropertiesOfNode(PropertyOperation.java:547) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.fillProperties(PropertyOperation.java:1634) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAndAddDataTypeByUid(PropertyOperation.java:1926) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypes(PropertyOperation.java:1874) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.replaceAllData(ApplicationDataTypeCache.java:312) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:248) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-05-04T01:40:56.279Z [main] INFO o.j.d.util.BackendOperation RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1670) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getChildrenNodes(JanusGraphGenericDao.java:1336) at org.openecomp.sdc.be.dao.janusgraph.HealingJanusGraphGenericDao.getChildrenNodes(HealingJanusGraphGenericDao.java:57) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getChildrenNodes(JanusGraphGenericDao.java:1390) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.findPropertiesOfNode(PropertyOperation.java:547) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.fillProperties(PropertyOperation.java:1634) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAndAddDataTypeByUid(PropertyOperation.java:1926) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypes(PropertyOperation.java:1874) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.getAllDataTypesFromGraph(ApplicationDataTypeCache.java:159) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.getAll(ApplicationDataTypeCache.java:170) at org.openecomp.sdc.be.tosca.PolicyExportParserImpl.getDataTypes(PolicyExportParserImpl.java:71) at org.openecomp.sdc.be.tosca.PolicyExportParserImpl.(PolicyExportParserImpl.java:67) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:204) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:309) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:294) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:40:57.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3c88f2b-1879-4364-a0bd-2fc85ee71713 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:57.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3c88f2b-1879-4364-a0bd-2fc85ee71713 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3411dc96 2021-05-04T01:40:57.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3c88f2b-1879-4364-a0bd-2fc85ee71713 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:57.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3c88f2b-1879-4364-a0bd-2fc85ee71713 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:40:58.081Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4d77d6-032c-468c-9a17-5cf88d0a294b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:40:58.081Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4d77d6-032c-468c-9a17-5cf88d0a294b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2819ffa7 2021-05-04T01:40:58.081Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4d77d6-032c-468c-9a17-5cf88d0a294b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:40:58.081Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4d77d6-032c-468c-9a17-5cf88d0a294b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:00.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eca35ce-40b9-46de-b8b4-0ad914a7cc2a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:00.506Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eca35ce-40b9-46de-b8b4-0ad914a7cc2a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41a00b09 2021-05-04T01:41:00.507Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eca35ce-40b9-46de-b8b4-0ad914a7cc2a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:00.508Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eca35ce-40b9-46de-b8b4-0ad914a7cc2a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a39f866-06e9-4e4a-9469-75d3d08c71ef PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a39f866-06e9-4e4a-9469-75d3d08c71ef PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1984311d 2021-05-04T01:41:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a39f866-06e9-4e4a-9469-75d3d08c71ef PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:03.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a39f866-06e9-4e4a-9469-75d3d08c71ef PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:06.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c796b8d8-11f7-4082-a042-cd75c7b3aef0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:06.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c796b8d8-11f7-4082-a042-cd75c7b3aef0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23cbabd7 2021-05-04T01:41:06.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c796b8d8-11f7-4082-a042-cd75c7b3aef0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:06.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c796b8d8-11f7-4082-a042-cd75c7b3aef0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:10.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed01441e-5e5a-43c8-8c9d-fa2170a4cf46 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:10.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed01441e-5e5a-43c8-8c9d-fa2170a4cf46 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44c89cce 2021-05-04T01:41:10.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed01441e-5e5a-43c8-8c9d-fa2170a4cf46 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:10.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed01441e-5e5a-43c8-8c9d-fa2170a4cf46 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:12.181Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00df6513-d871-48ec-9a78-e109467366c4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:12.181Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00df6513-d871-48ec-9a78-e109467366c4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5712f1d 2021-05-04T01:41:12.181Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00df6513-d871-48ec-9a78-e109467366c4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:12.181Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00df6513-d871-48ec-9a78-e109467366c4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:13.007Z [main] INFO o.o.s.b.d.c.SdcSchemaFilesCassandraDao RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** SdcSchemaFilesCassandraDao created 2021-05-04T01:41:13.267Z [main] WARN o.o.s.b.c.s.CertificateManagerImpl RequestID=4c7f4bbf-091e-40dd-b569-605e5e873e36 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Environment variable 'SDC_CERT_DIR' was not provided. Could not load certificates. 2021-05-04T01:41:15.802Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4ccbed9-d4cd-4ca2-944d-83bd038c1729 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:15.802Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4ccbed9-d4cd-4ca2-944d-83bd038c1729 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d45fa77 2021-05-04T01:41:15.802Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4ccbed9-d4cd-4ca2-944d-83bd038c1729 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:15.802Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4ccbed9-d4cd-4ca2-944d-83bd038c1729 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:16.103Z [main] INFO o.o.s.b.c.d.e.EnvironmentsEngine RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE Environments engine is disabled 2021-05-04T01:41:16.180Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:41:16.188Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:41:16.189Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:16.192Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:41:16.192Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:16.193Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:41:16.193Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:16.195Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:41:16.222Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:41:16.223Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:41:18.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8605cd6f-d29a-40b1-972a-35c6c5b65a92 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:18.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8605cd6f-d29a-40b1-972a-35c6c5b65a92 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@432e7b1b 2021-05-04T01:41:18.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8605cd6f-d29a-40b1-972a-35c6c5b65a92 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:18.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8605cd6f-d29a-40b1-972a-35c6c5b65a92 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:19.080Z [ApplicationDataTypeCacheThread-1] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL ErrorCategory=INFO RequestID=null ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=500 TargetEntity=ReplaceDataTypesCache Internal flow error. Operation: Succeed to replace the data types cache 2021-05-04T01:41:20.251Z [main] INFO o.o.s.b.c.s.AsdcComponentsCleanerTask RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE ServerIPAddress=10.242.235.154 ServerFQDN=dev-sdc-be-69f59769db-d8vzh Enter init method of AsdcComponentsCleaner 2021-05-04T01:41:20.252Z [main] INFO o.o.s.b.c.s.AsdcComponentsCleanerTask RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE ServerIPAddress=10.242.235.154 ServerFQDN=dev-sdc-be-69f59769db-d8vzh End init method of AsdcComponentsCleaner 2021-05-04T01:41:20.366Z [main] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE ServerIPAddress=10.242.235.154 ServerFQDN=dev-sdc-be-69f59769db-d8vzh Autowired annotation is not supported on static methods: public static void org.openecomp.sdc.be.components.impl.ImportUtils.setComponentsUtils(org.openecomp.sdc.be.impl.ComponentsUtils) 2021-05-04T01:41:21.225Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:41:21.230Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:41:21.230Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:21.232Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:41:21.232Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:21.233Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:41:21.233Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:21.235Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:41:21.235Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:41:21.236Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:41:22.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=964c16f8-9a0a-4ebf-9397-f7a12bcffaa3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:22.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=964c16f8-9a0a-4ebf-9397-f7a12bcffaa3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4300a15c 2021-05-04T01:41:22.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=964c16f8-9a0a-4ebf-9397-f7a12bcffaa3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:22.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=964c16f8-9a0a-4ebf-9397-f7a12bcffaa3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:22.822Z [main] INFO o.o.s.b.d.cassandra.FeatureToggleDao RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE ServerIPAddress=10.242.235.154 ServerFQDN=dev-sdc-be-69f59769db-d8vzh ** FeatureToggleDao created 2021-05-04T01:41:23.171Z [main] INFO o.s.web.context.ContextLoader RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE ServerIPAddress=10.242.235.154 ServerFQDN=dev-sdc-be-69f59769db-d8vzh Root WebApplicationContext initialized in 75672 ms 2021-05-04T01:41:24.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=800f6737-8644-4265-9497-061e403328aa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:24.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=800f6737-8644-4265-9497-061e403328aa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f8577e6 2021-05-04T01:41:24.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=800f6737-8644-4265-9497-061e403328aa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:24.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=800f6737-8644-4265-9497-061e403328aa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:26.744Z [main] INFO o.a.c.b.FluentPropertyBeanIntrospector RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE ServerIPAddress=10.242.235.154 ServerFQDN=dev-sdc-be-69f59769db-d8vzh Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property. 2021-05-04T01:41:26.768Z [main] INFO o.a.c.b.FluentPropertyBeanIntrospector RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE ServerIPAddress=10.242.235.154 ServerFQDN=dev-sdc-be-69f59769db-d8vzh Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property. 2021-05-04T01:41:26.797Z [main] INFO o.a.c.b.FluentPropertyBeanIntrospector RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE ServerIPAddress=10.242.235.154 ServerFQDN=dev-sdc-be-69f59769db-d8vzh Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property. 2021-05-04T01:41:28.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adc5a08b-1e92-45bb-bc70-233dd1ac33c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:28.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adc5a08b-1e92-45bb-bc70-233dd1ac33c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d01aa23 2021-05-04T01:41:28.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adc5a08b-1e92-45bb-bc70-233dd1ac33c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:28.217Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adc5a08b-1e92-45bb-bc70-233dd1ac33c6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:29.832Z [main] INFO o.h.validator.internal.util.Version RequestID=28affea3-4fb0-4fed-bde7-0a02e20883c6 PartnerName=SDC.BE ServerIPAddress=10.242.235.154 ServerFQDN=dev-sdc-be-69f59769db-d8vzh HV000001: Hibernate Validator 5.3.6.Final 2021-05-04T01:41:30.440Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17fa7645-56a9-4731-a2a1-ed3c232aea80 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:30.440Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17fa7645-56a9-4731-a2a1-ed3c232aea80 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d47f43b 2021-05-04T01:41:30.440Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17fa7645-56a9-4731-a2a1-ed3c232aea80 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:30.440Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17fa7645-56a9-4731-a2a1-ed3c232aea80 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:31.238Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:41:31.240Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:41:31.241Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:31.245Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:41:31.245Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:31.247Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:41:31.248Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:31.250Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:41:31.264Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:41:31.264Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:41:32.106Z [qtp580718781-15] INFO com.jcabi.aspects.aj.NamedThreads RequestID=null ServerIPAddress=10.242.235.154 ServiceName=/healthCheck EntryTimestamp=2021-05-04T01:41:32.002Z InvocationID=a2a752c8-974e-426b-86c7-2a74956a26c5 PartnerName=Apache-HttpClient/4.5.3 (Java/11.0.8) auditOn=true ServerFQDN=dev-sdc-be-69f59769db-d8vzh jcabi-aspects 0.20.1/f0aff93 started new daemon thread jcabi-loggable for watching of @Loggable annotated methods 2021-05-04T01:41:33.383Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25eb1abb-c304-4d84-ae9e-448fee2953c0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:33.383Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25eb1abb-c304-4d84-ae9e-448fee2953c0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b8ba28e 2021-05-04T01:41:33.383Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25eb1abb-c304-4d84-ae9e-448fee2953c0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:33.383Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25eb1abb-c304-4d84-ae9e-448fee2953c0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:36.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4891e18f-7168-4a97-a768-6739c8b1a266 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:36.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4891e18f-7168-4a97-a768-6739c8b1a266 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4fd5da5e 2021-05-04T01:41:36.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4891e18f-7168-4a97-a768-6739c8b1a266 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:36.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4891e18f-7168-4a97-a768-6739c8b1a266 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:39.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a2e84f1-cdda-4433-87f3-5a2230248018 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:39.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a2e84f1-cdda-4433-87f3-5a2230248018 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1fd53135 2021-05-04T01:41:39.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a2e84f1-cdda-4433-87f3-5a2230248018 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:39.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a2e84f1-cdda-4433-87f3-5a2230248018 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:42.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9c6cef0-d1f0-4be4-8247-788916a9cba4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:42.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9c6cef0-d1f0-4be4-8247-788916a9cba4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4de1f8d8 2021-05-04T01:41:42.409Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9c6cef0-d1f0-4be4-8247-788916a9cba4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:42.410Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9c6cef0-d1f0-4be4-8247-788916a9cba4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:45.634Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71fcecd7-1307-498c-b93e-fa9d4b99b5a9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:45.634Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71fcecd7-1307-498c-b93e-fa9d4b99b5a9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b08ac38 2021-05-04T01:41:45.634Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71fcecd7-1307-498c-b93e-fa9d4b99b5a9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:45.634Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71fcecd7-1307-498c-b93e-fa9d4b99b5a9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:48.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=379baaf9-8bef-4487-aa18-9f44f25603b6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:48.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=379baaf9-8bef-4487-aa18-9f44f25603b6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5aee962d 2021-05-04T01:41:48.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=379baaf9-8bef-4487-aa18-9f44f25603b6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:48.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=379baaf9-8bef-4487-aa18-9f44f25603b6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:51.269Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:41:51.274Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:41:51.274Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:51.276Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:41:51.276Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:51.278Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:41:51.278Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:41:51.280Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:41:51.281Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:41:51.282Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:41:51.529Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d65af30b-8739-4f2f-abf6-606b3e2c9e16 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:51.529Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d65af30b-8739-4f2f-abf6-606b3e2c9e16 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c0458dc 2021-05-04T01:41:51.529Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d65af30b-8739-4f2f-abf6-606b3e2c9e16 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:51.529Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d65af30b-8739-4f2f-abf6-606b3e2c9e16 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:54.792Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c661e65-3e39-4e09-ab84-011c55121a05 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:54.792Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c661e65-3e39-4e09-ab84-011c55121a05 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ee15abc 2021-05-04T01:41:54.792Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c661e65-3e39-4e09-ab84-011c55121a05 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:54.792Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c661e65-3e39-4e09-ab84-011c55121a05 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:41:57.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbb9a65b-a9a0-4673-ab92-e62cf2e6165a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:41:57.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbb9a65b-a9a0-4673-ab92-e62cf2e6165a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50765a34 2021-05-04T01:41:57.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbb9a65b-a9a0-4673-ab92-e62cf2e6165a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:41:57.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbb9a65b-a9a0-4673-ab92-e62cf2e6165a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:00.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=871a259f-5c53-45f3-986d-98e7f752a1a7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:00.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=871a259f-5c53-45f3-986d-98e7f752a1a7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e07e1e9 2021-05-04T01:42:00.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=871a259f-5c53-45f3-986d-98e7f752a1a7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:00.309Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=871a259f-5c53-45f3-986d-98e7f752a1a7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:03.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=802165e5-a78c-480c-82cc-d7b390b85a4c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:03.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=802165e5-a78c-480c-82cc-d7b390b85a4c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@300b844d 2021-05-04T01:42:03.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=802165e5-a78c-480c-82cc-d7b390b85a4c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:03.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=802165e5-a78c-480c-82cc-d7b390b85a4c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:06.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba4b0ca-37c7-4c2d-aec8-82f2691d73fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:06.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba4b0ca-37c7-4c2d-aec8-82f2691d73fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@141b0c4a 2021-05-04T01:42:06.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba4b0ca-37c7-4c2d-aec8-82f2691d73fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:06.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ba4b0ca-37c7-4c2d-aec8-82f2691d73fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:09.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d1b7fa-6da0-4316-880b-ee129c122367 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:09.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d1b7fa-6da0-4316-880b-ee129c122367 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76de5307 2021-05-04T01:42:09.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d1b7fa-6da0-4316-880b-ee129c122367 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:09.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d1b7fa-6da0-4316-880b-ee129c122367 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:11.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe4ef3d-1e49-4b4e-b14c-81a92d2a967f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:11.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe4ef3d-1e49-4b4e-b14c-81a92d2a967f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3717fc9c 2021-05-04T01:42:11.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe4ef3d-1e49-4b4e-b14c-81a92d2a967f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:11.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fe4ef3d-1e49-4b4e-b14c-81a92d2a967f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:15.389Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49d28fa0-ef9f-4550-b6df-9081a9ccdbb3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:15.390Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49d28fa0-ef9f-4550-b6df-9081a9ccdbb3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b522fdf 2021-05-04T01:42:15.390Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49d28fa0-ef9f-4550-b6df-9081a9ccdbb3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:15.390Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49d28fa0-ef9f-4550-b6df-9081a9ccdbb3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:18.189Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=412077b4-a66c-4063-a674-70b42c0badfa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:18.189Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=412077b4-a66c-4063-a674-70b42c0badfa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17e711a3 2021-05-04T01:42:18.189Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=412077b4-a66c-4063-a674-70b42c0badfa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:18.190Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=412077b4-a66c-4063-a674-70b42c0badfa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:20.907Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b36724dc-53de-48f9-b3e3-930b9a7b4342 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:20.908Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b36724dc-53de-48f9-b3e3-930b9a7b4342 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13253e33 2021-05-04T01:42:20.908Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b36724dc-53de-48f9-b3e3-930b9a7b4342 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:20.910Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b36724dc-53de-48f9-b3e3-930b9a7b4342 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:24.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38b49ca4-371d-4434-8bd1-0462f040efb9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:24.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38b49ca4-371d-4434-8bd1-0462f040efb9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bdec3d7 2021-05-04T01:42:24.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38b49ca4-371d-4434-8bd1-0462f040efb9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:24.402Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38b49ca4-371d-4434-8bd1-0462f040efb9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:27.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49a0a64-4cc1-4b51-9e08-e6a8d96d06e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:27.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49a0a64-4cc1-4b51-9e08-e6a8d96d06e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51b82ba4 2021-05-04T01:42:27.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49a0a64-4cc1-4b51-9e08-e6a8d96d06e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:27.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49a0a64-4cc1-4b51-9e08-e6a8d96d06e6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:30.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e015e18-d806-4dcb-9f82-859985f0670f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:30.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e015e18-d806-4dcb-9f82-859985f0670f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7efd5239 2021-05-04T01:42:30.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e015e18-d806-4dcb-9f82-859985f0670f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:30.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e015e18-d806-4dcb-9f82-859985f0670f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:31.287Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:42:31.294Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:42:31.294Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:42:31.296Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:42:31.296Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:42:31.300Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:42:31.300Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:42:31.302Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:42:31.303Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:42:31.307Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:42:33.447Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ac2a103-7b0f-444c-808f-530509dfc026 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:33.447Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ac2a103-7b0f-444c-808f-530509dfc026 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44933f73 2021-05-04T01:42:33.448Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ac2a103-7b0f-444c-808f-530509dfc026 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:33.449Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ac2a103-7b0f-444c-808f-530509dfc026 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:36.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003d4db6-d76b-4d06-82fa-d5cb630e2a42 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:36.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003d4db6-d76b-4d06-82fa-d5cb630e2a42 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@293e8c4d 2021-05-04T01:42:36.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003d4db6-d76b-4d06-82fa-d5cb630e2a42 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:36.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=003d4db6-d76b-4d06-82fa-d5cb630e2a42 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:39.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8876b40c-7323-4333-aefe-72f870603b4d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:39.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8876b40c-7323-4333-aefe-72f870603b4d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39cb67c5 2021-05-04T01:42:39.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8876b40c-7323-4333-aefe-72f870603b4d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:39.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8876b40c-7323-4333-aefe-72f870603b4d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:42.602Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98bebf40-b464-4773-bb8a-a9cc3dc8e2df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:42.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98bebf40-b464-4773-bb8a-a9cc3dc8e2df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2db3965f 2021-05-04T01:42:42.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98bebf40-b464-4773-bb8a-a9cc3dc8e2df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:42.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98bebf40-b464-4773-bb8a-a9cc3dc8e2df PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:45.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56b3caf2-9a68-4ad1-afbd-6495faecfc89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:45.703Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56b3caf2-9a68-4ad1-afbd-6495faecfc89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70065c3d 2021-05-04T01:42:45.704Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56b3caf2-9a68-4ad1-afbd-6495faecfc89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:45.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56b3caf2-9a68-4ad1-afbd-6495faecfc89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:54.449Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c724eda3-c35f-4c2b-a228-4b121f0fe39c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:54.449Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c724eda3-c35f-4c2b-a228-4b121f0fe39c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d87fa94 2021-05-04T01:42:54.449Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c724eda3-c35f-4c2b-a228-4b121f0fe39c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:54.450Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c724eda3-c35f-4c2b-a228-4b121f0fe39c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:55.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c57f841f-f97d-458a-9e37-9c6195aca474 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:55.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c57f841f-f97d-458a-9e37-9c6195aca474 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f1045ae 2021-05-04T01:42:55.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c57f841f-f97d-458a-9e37-9c6195aca474 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:55.401Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c57f841f-f97d-458a-9e37-9c6195aca474 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:56.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0d388e3-0ac0-4331-b244-05725e416ca7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:56.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0d388e3-0ac0-4331-b244-05725e416ca7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ea0237d 2021-05-04T01:42:56.301Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0d388e3-0ac0-4331-b244-05725e416ca7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:56.302Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0d388e3-0ac0-4331-b244-05725e416ca7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:42:57.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10a2a66d-f52e-4beb-87ab-350c85a438a7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:42:57.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10a2a66d-f52e-4beb-87ab-350c85a438a7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c856f20 2021-05-04T01:42:57.501Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10a2a66d-f52e-4beb-87ab-350c85a438a7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:42:57.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10a2a66d-f52e-4beb-87ab-350c85a438a7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:01.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bff436dd-0adb-42b2-910f-1af0979392c7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:01.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bff436dd-0adb-42b2-910f-1af0979392c7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ec4617e 2021-05-04T01:43:01.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bff436dd-0adb-42b2-910f-1af0979392c7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:01.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bff436dd-0adb-42b2-910f-1af0979392c7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e39a9ea9-8d34-4b18-a649-a58a88cf2192 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e39a9ea9-8d34-4b18-a649-a58a88cf2192 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63346c3d 2021-05-04T01:43:03.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e39a9ea9-8d34-4b18-a649-a58a88cf2192 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:03.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e39a9ea9-8d34-4b18-a649-a58a88cf2192 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:06.477Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=143d02bc-e024-4212-a12e-8caba1379fa3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:06.477Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=143d02bc-e024-4212-a12e-8caba1379fa3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c7000ed 2021-05-04T01:43:06.477Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=143d02bc-e024-4212-a12e-8caba1379fa3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:06.477Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=143d02bc-e024-4212-a12e-8caba1379fa3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:10.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33e51864-b5d4-4f10-b7bd-17ec340a3d03 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:10.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33e51864-b5d4-4f10-b7bd-17ec340a3d03 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b9b902c 2021-05-04T01:43:10.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33e51864-b5d4-4f10-b7bd-17ec340a3d03 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:10.254Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33e51864-b5d4-4f10-b7bd-17ec340a3d03 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:11.813Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa9022c5-f05c-4033-a856-b735427c0ff3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:11.813Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa9022c5-f05c-4033-a856-b735427c0ff3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@659cfb0 2021-05-04T01:43:11.813Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa9022c5-f05c-4033-a856-b735427c0ff3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:11.813Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa9022c5-f05c-4033-a856-b735427c0ff3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:15.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a6efd28-657d-4ee3-9cac-a6aa73382e6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:15.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a6efd28-657d-4ee3-9cac-a6aa73382e6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@464c1ec2 2021-05-04T01:43:15.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a6efd28-657d-4ee3-9cac-a6aa73382e6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:15.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a6efd28-657d-4ee3-9cac-a6aa73382e6a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:18.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3da8d663-b875-4f4e-b83e-bfbace327f63 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:18.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3da8d663-b875-4f4e-b83e-bfbace327f63 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5aa7098e 2021-05-04T01:43:18.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3da8d663-b875-4f4e-b83e-bfbace327f63 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:18.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3da8d663-b875-4f4e-b83e-bfbace327f63 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:21.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df4ee711-475d-45ae-8337-7c373a2e6532 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:21.203Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df4ee711-475d-45ae-8337-7c373a2e6532 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d6af806 2021-05-04T01:43:21.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df4ee711-475d-45ae-8337-7c373a2e6532 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:21.204Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df4ee711-475d-45ae-8337-7c373a2e6532 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:28.610Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:43:28.611Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:43:30.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c39aba29-c4ca-4319-9f74-e71fc60f86fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:30.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c39aba29-c4ca-4319-9f74-e71fc60f86fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45869254 2021-05-04T01:43:30.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c39aba29-c4ca-4319-9f74-e71fc60f86fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:30.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c39aba29-c4ca-4319-9f74-e71fc60f86fa PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:43:30.667Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c39aba29-c4ca-4319-9f74-e71fc60f86fa ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:43:30.668Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=c39aba29-c4ca-4319-9f74-e71fc60f86fa ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:43:31.288Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:43:31.295Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:43:31.296Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:43:31.297Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:43:31.297Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:43:31.299Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:43:31.299Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:43:31.301Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:43:31.302Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:43:31.303Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:43:31.353Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf0c0801-365e-431f-9c53-0b73d9a8daae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:31.353Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf0c0801-365e-431f-9c53-0b73d9a8daae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@631a3e28 2021-05-04T01:43:31.353Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf0c0801-365e-431f-9c53-0b73d9a8daae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:31.353Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf0c0801-365e-431f-9c53-0b73d9a8daae PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:43:31.360Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=cf0c0801-365e-431f-9c53-0b73d9a8daae ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connection reset 2021-05-04T01:43:31.362Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=cf0c0801-365e-431f-9c53-0b73d9a8daae ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connection reset at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: javax.net.ssl.SSLException: Connection reset at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.handleException(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Suppressed: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source) at java.base/java.net.SocketOutputStream.write(Unknown Source) at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(Unknown Source) ... 35 common frames omitted Caused by: java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) ... 31 common frames omitted 2021-05-04T01:43:32.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fb322f7-2f61-43a9-b463-fadae0270863 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:32.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fb322f7-2f61-43a9-b463-fadae0270863 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@113b0bc 2021-05-04T01:43:32.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fb322f7-2f61-43a9-b463-fadae0270863 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:32.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fb322f7-2f61-43a9-b463-fadae0270863 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:32.879Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8fb322f7-2f61-43a9-b463-fadae0270863 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T01:43:32.888Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8fb322f7-2f61-43a9-b463-fadae0270863 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:43:32.889Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8fb322f7-2f61-43a9-b463-fadae0270863 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:43:32.890Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=8fb322f7-2f61-43a9-b463-fadae0270863 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:43:42.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa77d0cd-8139-4dfb-aa45-7e82f2421131 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:42.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa77d0cd-8139-4dfb-aa45-7e82f2421131 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d5aaab 2021-05-04T01:43:42.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa77d0cd-8139-4dfb-aa45-7e82f2421131 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:42.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa77d0cd-8139-4dfb-aa45-7e82f2421131 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:45.484Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=aa77d0cd-8139-4dfb-aa45-7e82f2421131 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:43:45.486Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=aa77d0cd-8139-4dfb-aa45-7e82f2421131 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:43:46.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fafad672-be64-45b3-b39a-1bd7b88b0b2c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:46.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fafad672-be64-45b3-b39a-1bd7b88b0b2c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c733ee 2021-05-04T01:43:46.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fafad672-be64-45b3-b39a-1bd7b88b0b2c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:46.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fafad672-be64-45b3-b39a-1bd7b88b0b2c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:49.174Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=fafad672-be64-45b3-b39a-1bd7b88b0b2c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:43:49.174Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=fafad672-be64-45b3-b39a-1bd7b88b0b2c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:43:49.712Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e030df3a-3fe5-4b8f-ad4f-09e97313ef46 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:49.712Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e030df3a-3fe5-4b8f-ad4f-09e97313ef46 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63214bb8 2021-05-04T01:43:49.712Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e030df3a-3fe5-4b8f-ad4f-09e97313ef46 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:49.712Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e030df3a-3fe5-4b8f-ad4f-09e97313ef46 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:52.726Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e030df3a-3fe5-4b8f-ad4f-09e97313ef46 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:43:52.727Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e030df3a-3fe5-4b8f-ad4f-09e97313ef46 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:43:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40272efe-d587-4848-a099-ad2fb00ef00d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:53.502Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40272efe-d587-4848-a099-ad2fb00ef00d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55aaa1aa 2021-05-04T01:43:53.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40272efe-d587-4848-a099-ad2fb00ef00d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:53.503Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40272efe-d587-4848-a099-ad2fb00ef00d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:43:56.519Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=40272efe-d587-4848-a099-ad2fb00ef00d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T01:43:56.520Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=40272efe-d587-4848-a099-ad2fb00ef00d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T01:43:57.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74f6a0c8-77b1-494a-8d61-7f3f8947bcd8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:43:57.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74f6a0c8-77b1-494a-8d61-7f3f8947bcd8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@373a2a5e 2021-05-04T01:43:57.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74f6a0c8-77b1-494a-8d61-7f3f8947bcd8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:43:57.304Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74f6a0c8-77b1-494a-8d61-7f3f8947bcd8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:02.576Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=74f6a0c8-77b1-494a-8d61-7f3f8947bcd8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:44:02.576Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=74f6a0c8-77b1-494a-8d61-7f3f8947bcd8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:44:03.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6b27a54-1150-43b5-8bbd-ccd4a40cb640 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:03.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6b27a54-1150-43b5-8bbd-ccd4a40cb640 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@fd24b75 2021-05-04T01:44:03.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6b27a54-1150-43b5-8bbd-ccd4a40cb640 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:03.516Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6b27a54-1150-43b5-8bbd-ccd4a40cb640 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:04.238Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c6b27a54-1150-43b5-8bbd-ccd4a40cb640 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:44:04.238Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=c6b27a54-1150-43b5-8bbd-ccd4a40cb640 ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:44:05.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d7a6965-bcba-4ba0-9c26-5b24f6d8c10c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:05.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d7a6965-bcba-4ba0-9c26-5b24f6d8c10c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bad6e50 2021-05-04T01:44:05.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d7a6965-bcba-4ba0-9c26-5b24f6d8c10c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:05.103Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9d7a6965-bcba-4ba0-9c26-5b24f6d8c10c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:06.650Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5d10359-ed5e-4287-88e0-edd7d64fedc5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:06.651Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5d10359-ed5e-4287-88e0-edd7d64fedc5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c7a7245 2021-05-04T01:44:06.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5d10359-ed5e-4287-88e0-edd7d64fedc5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:06.652Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5d10359-ed5e-4287-88e0-edd7d64fedc5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:07.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40a2cdf5-6e19-412c-9218-02efba29d5b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:07.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40a2cdf5-6e19-412c-9218-02efba29d5b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a669731 2021-05-04T01:44:07.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40a2cdf5-6e19-412c-9218-02efba29d5b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:07.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40a2cdf5-6e19-412c-9218-02efba29d5b4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:08.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=733c5793-5754-4948-8ada-8239a85b3b98 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:08.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=733c5793-5754-4948-8ada-8239a85b3b98 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e21cc77 2021-05-04T01:44:08.407Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=733c5793-5754-4948-8ada-8239a85b3b98 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:08.408Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=733c5793-5754-4948-8ada-8239a85b3b98 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:09.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23eee955-f73c-4937-a86b-c801833a1eeb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:09.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23eee955-f73c-4937-a86b-c801833a1eeb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@415675ac 2021-05-04T01:44:09.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23eee955-f73c-4937-a86b-c801833a1eeb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:09.300Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23eee955-f73c-4937-a86b-c801833a1eeb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:10.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5a2ac73-e0ac-48dc-aeaf-78732beae8bf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:10.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5a2ac73-e0ac-48dc-aeaf-78732beae8bf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50dfe830 2021-05-04T01:44:10.101Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5a2ac73-e0ac-48dc-aeaf-78732beae8bf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:10.102Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5a2ac73-e0ac-48dc-aeaf-78732beae8bf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:11.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54cf7b17-136e-47e8-a93b-3da3194d4fd9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:11.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54cf7b17-136e-47e8-a93b-3da3194d4fd9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@674d1530 2021-05-04T01:44:11.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54cf7b17-136e-47e8-a93b-3da3194d4fd9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:11.100Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54cf7b17-136e-47e8-a93b-3da3194d4fd9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:14.109Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=54cf7b17-136e-47e8-a93b-3da3194d4fd9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:44:14.111Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=54cf7b17-136e-47e8-a93b-3da3194d4fd9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:44:14.113Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=54cf7b17-136e-47e8-a93b-3da3194d4fd9 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:44:14.116Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=54cf7b17-136e-47e8-a93b-3da3194d4fd9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:44:19.129Z [cluster1-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:44:19.130Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46929273-2161-4de3-83ce-195f1a5ab66f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:19.130Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46929273-2161-4de3-83ce-195f1a5ab66f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28587208 2021-05-04T01:44:19.130Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46929273-2161-4de3-83ce-195f1a5ab66f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:19.131Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46929273-2161-4de3-83ce-195f1a5ab66f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:44:19.511Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=46929273-2161-4de3-83ce-195f1a5ab66f ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:44:19.512Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=46929273-2161-4de3-83ce-195f1a5ab66f ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:44:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11a3e6e4-ac19-41fe-91ee-7c23296f191c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11a3e6e4-ac19-41fe-91ee-7c23296f191c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f8bda5f 2021-05-04T01:44:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11a3e6e4-ac19-41fe-91ee-7c23296f191c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:20.200Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11a3e6e4-ac19-41fe-91ee-7c23296f191c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-05-04T01:44:21.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53e7946e-1f07-4ffd-9c78-951705ddbcfa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:21.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53e7946e-1f07-4ffd-9c78-951705ddbcfa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50e0b00b 2021-05-04T01:44:21.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53e7946e-1f07-4ffd-9c78-951705ddbcfa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:21.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53e7946e-1f07-4ffd-9c78-951705ddbcfa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:23.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6da70f9b-ef5b-42d0-80f8-c6aca28d8155 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:23.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6da70f9b-ef5b-42d0-80f8-c6aca28d8155 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3bc3402d 2021-05-04T01:44:23.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6da70f9b-ef5b-42d0-80f8-c6aca28d8155 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:23.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6da70f9b-ef5b-42d0-80f8-c6aca28d8155 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:24.545Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c29db61-2860-42b7-8910-05271d4b01f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:24.545Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c29db61-2860-42b7-8910-05271d4b01f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a48cf67 2021-05-04T01:44:24.545Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c29db61-2860-42b7-8910-05271d4b01f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:24.545Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c29db61-2860-42b7-8910-05271d4b01f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:27.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=941a9a3d-ba2d-4fc7-a0af-91b2559778b2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:27.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=941a9a3d-ba2d-4fc7-a0af-91b2559778b2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b90a753 2021-05-04T01:44:27.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=941a9a3d-ba2d-4fc7-a0af-91b2559778b2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:27.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=941a9a3d-ba2d-4fc7-a0af-91b2559778b2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:31.288Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:44:31.292Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:44:31.292Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:44:31.293Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:44:31.294Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:44:31.295Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:44:31.295Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:44:31.296Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:44:31.297Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:44:31.297Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:44:31.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8384f0a3-2b31-498c-9fdf-3ae00b94855f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:31.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8384f0a3-2b31-498c-9fdf-3ae00b94855f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ea52cc2 2021-05-04T01:44:31.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8384f0a3-2b31-498c-9fdf-3ae00b94855f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:31.949Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8384f0a3-2b31-498c-9fdf-3ae00b94855f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:32.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed1c38a8-a1c2-404e-9547-79164521dc5e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:32.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed1c38a8-a1c2-404e-9547-79164521dc5e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6fc6cf13 2021-05-04T01:44:32.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed1c38a8-a1c2-404e-9547-79164521dc5e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:32.902Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed1c38a8-a1c2-404e-9547-79164521dc5e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:39.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456f52b4-7be3-440f-9c7d-5ff83af0c7cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:39.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456f52b4-7be3-440f-9c7d-5ff83af0c7cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3329167b 2021-05-04T01:44:39.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456f52b4-7be3-440f-9c7d-5ff83af0c7cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:39.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=456f52b4-7be3-440f-9c7d-5ff83af0c7cb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:42.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3200f4fb-f2cd-4e95-8201-02ad9a0ca470 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:42.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3200f4fb-f2cd-4e95-8201-02ad9a0ca470 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d1d8572 2021-05-04T01:44:42.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3200f4fb-f2cd-4e95-8201-02ad9a0ca470 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:42.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3200f4fb-f2cd-4e95-8201-02ad9a0ca470 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:43.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87426e8a-f2d8-46be-97c7-e4b84d0c6019 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:43.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87426e8a-f2d8-46be-97c7-e4b84d0c6019 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ab96f2f 2021-05-04T01:44:43.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87426e8a-f2d8-46be-97c7-e4b84d0c6019 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:43.399Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87426e8a-f2d8-46be-97c7-e4b84d0c6019 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:46.405Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=87426e8a-f2d8-46be-97c7-e4b84d0c6019 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:44:46.406Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=87426e8a-f2d8-46be-97c7-e4b84d0c6019 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:44:46.407Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=87426e8a-f2d8-46be-97c7-e4b84d0c6019 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:44:46.407Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=87426e8a-f2d8-46be-97c7-e4b84d0c6019 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:44:46.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4f0f0fc-eb4c-40d6-9c8e-7684addb73e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:46.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4f0f0fc-eb4c-40d6-9c8e-7684addb73e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13de9f3c 2021-05-04T01:44:46.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4f0f0fc-eb4c-40d6-9c8e-7684addb73e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:46.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4f0f0fc-eb4c-40d6-9c8e-7684addb73e6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:47.160Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f4f0f0fc-eb4c-40d6-9c8e-7684addb73e6 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:44:47.160Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f4f0f0fc-eb4c-40d6-9c8e-7684addb73e6 ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:44:48.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2670f123-e5e4-4575-9c6e-f4c9e32a37ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:48.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2670f123-e5e4-4575-9c6e-f4c9e32a37ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2948e792 2021-05-04T01:44:48.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2670f123-e5e4-4575-9c6e-f4c9e32a37ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:48.099Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2670f123-e5e4-4575-9c6e-f4c9e32a37ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:52.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac3df6ac-79f3-4b8a-8823-f5ae8b8c5eb7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:52.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac3df6ac-79f3-4b8a-8823-f5ae8b8c5eb7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28893583 2021-05-04T01:44:52.350Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac3df6ac-79f3-4b8a-8823-f5ae8b8c5eb7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:52.351Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac3df6ac-79f3-4b8a-8823-f5ae8b8c5eb7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:55.357Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=ac3df6ac-79f3-4b8a-8823-f5ae8b8c5eb7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:44:55.357Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=ac3df6ac-79f3-4b8a-8823-f5ae8b8c5eb7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:44:55.358Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ac3df6ac-79f3-4b8a-8823-f5ae8b8c5eb7 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:44:55.358Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=ac3df6ac-79f3-4b8a-8823-f5ae8b8c5eb7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:44:58.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6871e324-0ff3-494e-8da2-e04e02021add PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:58.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6871e324-0ff3-494e-8da2-e04e02021add PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f4968d1 2021-05-04T01:44:58.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6871e324-0ff3-494e-8da2-e04e02021add PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:58.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6871e324-0ff3-494e-8da2-e04e02021add PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:44:59.215Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6871e324-0ff3-494e-8da2-e04e02021add ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:44:59.216Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=6871e324-0ff3-494e-8da2-e04e02021add ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:44:59.999Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de6349b8-f5f8-4ecb-9f6e-644b465ef47e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:44:59.999Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de6349b8-f5f8-4ecb-9f6e-644b465ef47e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37b55043 2021-05-04T01:44:59.999Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de6349b8-f5f8-4ecb-9f6e-644b465ef47e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:44:59.999Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de6349b8-f5f8-4ecb-9f6e-644b465ef47e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:01.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfd973d-ff39-4596-951d-d21ed95a042e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:01.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfd973d-ff39-4596-951d-d21ed95a042e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56f15e1b 2021-05-04T01:45:01.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfd973d-ff39-4596-951d-d21ed95a042e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:01.251Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dcfd973d-ff39-4596-951d-d21ed95a042e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:04.153Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dc5f8c7-6d01-45ad-bcf3-549e3a352ece PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:04.153Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dc5f8c7-6d01-45ad-bcf3-549e3a352ece PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@527abbf4 2021-05-04T01:45:04.153Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dc5f8c7-6d01-45ad-bcf3-549e3a352ece PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:04.153Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dc5f8c7-6d01-45ad-bcf3-549e3a352ece PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:07.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e3f11c4-e57e-464b-8aad-f6bf91fb3cd1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:07.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e3f11c4-e57e-464b-8aad-f6bf91fb3cd1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71320ae8 2021-05-04T01:45:07.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e3f11c4-e57e-464b-8aad-f6bf91fb3cd1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:07.216Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e3f11c4-e57e-464b-8aad-f6bf91fb3cd1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:09.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26ae5e7-daed-4788-a1f0-56588022ecae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:09.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26ae5e7-daed-4788-a1f0-56588022ecae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c986bc5 2021-05-04T01:45:09.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26ae5e7-daed-4788-a1f0-56588022ecae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:09.303Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26ae5e7-daed-4788-a1f0-56588022ecae PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:12.298Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba88ec28-2383-499b-b69e-05b6cb98c429 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:12.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba88ec28-2383-499b-b69e-05b6cb98c429 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3045e1cb 2021-05-04T01:45:12.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba88ec28-2383-499b-b69e-05b6cb98c429 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:12.299Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba88ec28-2383-499b-b69e-05b6cb98c429 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:15.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f201dfaa-effb-4b1c-8a8b-66796911802b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:15.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f201dfaa-effb-4b1c-8a8b-66796911802b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37e7492c 2021-05-04T01:45:15.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f201dfaa-effb-4b1c-8a8b-66796911802b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:15.323Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f201dfaa-effb-4b1c-8a8b-66796911802b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:18.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=358ed56b-b672-4879-b668-7ef2430dc207 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:18.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=358ed56b-b672-4879-b668-7ef2430dc207 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56e37072 2021-05-04T01:45:18.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=358ed56b-b672-4879-b668-7ef2430dc207 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:18.199Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=358ed56b-b672-4879-b668-7ef2430dc207 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:21.207Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=358ed56b-b672-4879-b668-7ef2430dc207 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:45:21.208Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=358ed56b-b672-4879-b668-7ef2430dc207 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:45:21.209Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=358ed56b-b672-4879-b668-7ef2430dc207 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:45:21.209Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=358ed56b-b672-4879-b668-7ef2430dc207 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:45:26.221Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:45:26.222Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:45:31.292Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:45:31.299Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:45:31.299Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:45:31.301Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:45:31.301Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:45:31.303Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:45:31.303Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:45:31.305Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:45:31.306Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:45:31.307Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:45:32.229Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:45:32.229Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:45:32.230Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.235.132:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:45:32.231Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:45:39.241Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:45:39.243Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:45:39.302Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:45:39.302Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.235.132:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:45:40.516Z [cluster1-reconnection-1] ERROR c.d.driver.core.ControlConnection [Control connection] Cannot connect to any host, scheduling retry in 1000 milliseconds 2021-05-04T01:45:41.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6021fb03-53a8-4a92-abed-8ddd42981ed9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:41.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6021fb03-53a8-4a92-abed-8ddd42981ed9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@798aef28 2021-05-04T01:45:41.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6021fb03-53a8-4a92-abed-8ddd42981ed9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:41.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6021fb03-53a8-4a92-abed-8ddd42981ed9 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:45:44.808Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6021fb03-53a8-4a92-abed-8ddd42981ed9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:45:44.809Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6021fb03-53a8-4a92-abed-8ddd42981ed9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:45:44.809Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6021fb03-53a8-4a92-abed-8ddd42981ed9 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:45:44.810Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=6021fb03-53a8-4a92-abed-8ddd42981ed9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:45:47.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f00c4a-aabd-487f-8083-b6826455fa8e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:47.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f00c4a-aabd-487f-8083-b6826455fa8e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b7d89a5 2021-05-04T01:45:47.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f00c4a-aabd-487f-8083-b6826455fa8e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:47.274Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f00c4a-aabd-487f-8083-b6826455fa8e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:47.529Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e7f00c4a-aabd-487f-8083-b6826455fa8e ErrorCategory=INFO ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:45:47.532Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=e7f00c4a-aabd-487f-8083-b6826455fa8e ErrorCategory=ERROR ServiceName=BE Component CASSANDRA state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:45:48.977Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1bdcdb0-5286-44ff-97af-658e6fe38f15 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:48.977Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1bdcdb0-5286-44ff-97af-658e6fe38f15 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@80d4c05 2021-05-04T01:45:48.977Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1bdcdb0-5286-44ff-97af-658e6fe38f15 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:48.977Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1bdcdb0-5286-44ff-97af-658e6fe38f15 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:49.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83bdcdec-8be7-45aa-940b-a5d63b9c5340 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:49.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83bdcdec-8be7-45aa-940b-a5d63b9c5340 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ed6da25 2021-05-04T01:45:49.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83bdcdec-8be7-45aa-940b-a5d63b9c5340 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:49.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83bdcdec-8be7-45aa-940b-a5d63b9c5340 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:52.921Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=83bdcdec-8be7-45aa-940b-a5d63b9c5340 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:45:52.922Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=83bdcdec-8be7-45aa-940b-a5d63b9c5340 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:45:52.923Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=83bdcdec-8be7-45aa-940b-a5d63b9c5340 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:45:52.923Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=83bdcdec-8be7-45aa-940b-a5d63b9c5340 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:45:53.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c170b989-4acd-4ed9-823c-1b3b0ec97889 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:53.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c170b989-4acd-4ed9-823c-1b3b0ec97889 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@196158fb 2021-05-04T01:45:53.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c170b989-4acd-4ed9-823c-1b3b0ec97889 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:53.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c170b989-4acd-4ed9-823c-1b3b0ec97889 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:54.464Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c170b989-4acd-4ed9-823c-1b3b0ec97889 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:45:54.464Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=c170b989-4acd-4ed9-823c-1b3b0ec97889 ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:45:55.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f84b0e9-f36e-4dea-82ff-e3b4d6615317 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:55.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f84b0e9-f36e-4dea-82ff-e3b4d6615317 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d3a63dc 2021-05-04T01:45:55.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f84b0e9-f36e-4dea-82ff-e3b4d6615317 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:55.308Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f84b0e9-f36e-4dea-82ff-e3b4d6615317 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:56.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=774450b8-44ad-4291-944e-f52f2e55e32c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:56.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=774450b8-44ad-4291-944e-f52f2e55e32c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75158dac 2021-05-04T01:45:56.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=774450b8-44ad-4291-944e-f52f2e55e32c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:56.400Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=774450b8-44ad-4291-944e-f52f2e55e32c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:57.701Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8753f5f-5fb9-48a2-b3b2-20c505be96cc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:57.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8753f5f-5fb9-48a2-b3b2-20c505be96cc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66a0b575 2021-05-04T01:45:57.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8753f5f-5fb9-48a2-b3b2-20c505be96cc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:57.705Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8753f5f-5fb9-48a2-b3b2-20c505be96cc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:45:58.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4380c2f7-2c57-4d40-9543-564f99ea123f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:45:58.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4380c2f7-2c57-4d40-9543-564f99ea123f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72eb43df 2021-05-04T01:45:58.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4380c2f7-2c57-4d40-9543-564f99ea123f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:45:58.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4380c2f7-2c57-4d40-9543-564f99ea123f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:00.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbdd5814-9052-4374-afaa-01ff24836354 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:00.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbdd5814-9052-4374-afaa-01ff24836354 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cc1eadd 2021-05-04T01:46:00.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbdd5814-9052-4374-afaa-01ff24836354 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:00.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbdd5814-9052-4374-afaa-01ff24836354 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:01.202Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1289ab25-9d59-4188-ace3-962fc5668ab5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:01.207Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1289ab25-9d59-4188-ace3-962fc5668ab5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25e07004 2021-05-04T01:46:01.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1289ab25-9d59-4188-ace3-962fc5668ab5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:01.208Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1289ab25-9d59-4188-ace3-962fc5668ab5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:03.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9dc8f5a-0d88-4fb0-800e-07991c090777 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:03.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9dc8f5a-0d88-4fb0-800e-07991c090777 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@460862d0 2021-05-04T01:46:03.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9dc8f5a-0d88-4fb0-800e-07991c090777 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:03.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9dc8f5a-0d88-4fb0-800e-07991c090777 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:06.000Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbb0b990-a9c0-4eef-a5dd-c33754c876c4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:06.001Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbb0b990-a9c0-4eef-a5dd-c33754c876c4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ed53475 2021-05-04T01:46:06.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbb0b990-a9c0-4eef-a5dd-c33754c876c4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:06.002Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbb0b990-a9c0-4eef-a5dd-c33754c876c4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:09.007Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=dbb0b990-a9c0-4eef-a5dd-c33754c876c4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:46:09.008Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=dbb0b990-a9c0-4eef-a5dd-c33754c876c4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:46:09.009Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=dbb0b990-a9c0-4eef-a5dd-c33754c876c4 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:46:09.010Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=dbb0b990-a9c0-4eef-a5dd-c33754c876c4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:46:14.031Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:46:14.032Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:46:15.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fedfb6a3-c347-4099-86d4-bbf530ace29d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:15.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fedfb6a3-c347-4099-86d4-bbf530ace29d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d31bc70 2021-05-04T01:46:15.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fedfb6a3-c347-4099-86d4-bbf530ace29d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:15.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fedfb6a3-c347-4099-86d4-bbf530ace29d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:17.762Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fedfb6a3-c347-4099-86d4-bbf530ace29d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:46:17.764Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=fedfb6a3-c347-4099-86d4-bbf530ace29d ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:46:18.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24386dc6-e183-4c98-8eab-7a2ce18569f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:18.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24386dc6-e183-4c98-8eab-7a2ce18569f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3eec74de 2021-05-04T01:46:18.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24386dc6-e183-4c98-8eab-7a2ce18569f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:18.497Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24386dc6-e183-4c98-8eab-7a2ce18569f1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:20.253Z [ComponentsCleanThread-1] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null lock resource with name DELETE_LOCKER 2021-05-04T01:46:21.504Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=24386dc6-e183-4c98-8eab-7a2ce18569f1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:46:21.504Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=24386dc6-e183-4c98-8eab-7a2ce18569f1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:46:21.505Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=24386dc6-e183-4c98-8eab-7a2ce18569f1 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:46:21.506Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=24386dc6-e183-4c98-8eab-7a2ce18569f1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:46:22.204Z [ComponentsCleanThread-1] INFO o.o.s.b.c.s.ComponentsCleanBusinessLogic RequestID=null Lock cleanup operation is done with result = OK 2021-05-04T01:46:22.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d62c871-9b8c-4ba1-a044-bb84e19772da PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:22.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d62c871-9b8c-4ba1-a044-bb84e19772da PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79dd99ea 2021-05-04T01:46:22.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d62c871-9b8c-4ba1-a044-bb84e19772da PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:22.801Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d62c871-9b8c-4ba1-a044-bb84e19772da PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:25.843Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7d62c871-9b8c-4ba1-a044-bb84e19772da ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:46:25.844Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7d62c871-9b8c-4ba1-a044-bb84e19772da ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:46:30.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c46bf8b-fa8c-41ec-93d5-60e4480c77f6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:30.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c46bf8b-fa8c-41ec-93d5-60e4480c77f6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e9371b3 2021-05-04T01:46:30.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c46bf8b-fa8c-41ec-93d5-60e4480c77f6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:30.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c46bf8b-fa8c-41ec-93d5-60e4480c77f6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:31.286Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:46:31.290Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:46:31.290Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:46:31.292Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:46:31.292Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:46:31.294Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:46:31.294Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:46:31.296Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error executing HTTP request. message-router.onap:3904 failed to respond; blacklisting for 2 minutes 2021-05-04T01:46:31.296Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:46:31.297Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:46:33.623Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6c46bf8b-fa8c-41ec-93d5-60e4480c77f6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:46:33.623Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6c46bf8b-fa8c-41ec-93d5-60e4480c77f6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:46:38.253Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4b0364c-c2be-460f-b681-9286b7c4dcaa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:38.253Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4b0364c-c2be-460f-b681-9286b7c4dcaa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26036f68 2021-05-04T01:46:38.253Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4b0364c-c2be-460f-b681-9286b7c4dcaa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:38.253Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4b0364c-c2be-460f-b681-9286b7c4dcaa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:39.027Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c4b0364c-c2be-460f-b681-9286b7c4dcaa ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:46:39.027Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=c4b0364c-c2be-460f-b681-9286b7c4dcaa ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:46:39.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28518217-5350-416a-a09f-ebb126fa607d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:39.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28518217-5350-416a-a09f-ebb126fa607d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41a1c176 2021-05-04T01:46:39.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28518217-5350-416a-a09f-ebb126fa607d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:39.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28518217-5350-416a-a09f-ebb126fa607d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:42.809Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=28518217-5350-416a-a09f-ebb126fa607d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:46:42.810Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=28518217-5350-416a-a09f-ebb126fa607d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:46:42.810Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=28518217-5350-416a-a09f-ebb126fa607d ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:46:42.811Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=28518217-5350-416a-a09f-ebb126fa607d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:46:44.748Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50ee2e7f-c035-48cc-8c22-4d671f0f237d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:44.748Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50ee2e7f-c035-48cc-8c22-4d671f0f237d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57a2a613 2021-05-04T01:46:44.748Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50ee2e7f-c035-48cc-8c22-4d671f0f237d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:44.748Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50ee2e7f-c035-48cc-8c22-4d671f0f237d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:47.800Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=50ee2e7f-c035-48cc-8c22-4d671f0f237d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:46:47.801Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=50ee2e7f-c035-48cc-8c22-4d671f0f237d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:46:48.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8b1de0f-ba5f-48ff-afb7-eeccae7a631e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:48.599Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8b1de0f-ba5f-48ff-afb7-eeccae7a631e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@fcd3151 2021-05-04T01:46:48.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8b1de0f-ba5f-48ff-afb7-eeccae7a631e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:48.600Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8b1de0f-ba5f-48ff-afb7-eeccae7a631e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:49.506Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c8b1de0f-ba5f-48ff-afb7-eeccae7a631e ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:46:49.507Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=c8b1de0f-ba5f-48ff-afb7-eeccae7a631e ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:46:51.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cea7d530-f555-49e7-b5c4-2a3d7d8bcf95 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:51.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cea7d530-f555-49e7-b5c4-2a3d7d8bcf95 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f85f97 2021-05-04T01:46:51.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cea7d530-f555-49e7-b5c4-2a3d7d8bcf95 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:51.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cea7d530-f555-49e7-b5c4-2a3d7d8bcf95 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:46:54.471Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=cea7d530-f555-49e7-b5c4-2a3d7d8bcf95 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:46:54.471Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=cea7d530-f555-49e7-b5c4-2a3d7d8bcf95 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:46:54.472Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cea7d530-f555-49e7-b5c4-2a3d7d8bcf95 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:46:54.472Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=cea7d530-f555-49e7-b5c4-2a3d7d8bcf95 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:46:58.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e01a47db-fb63-4954-8a22-fc2b37a03e7c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:46:58.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e01a47db-fb63-4954-8a22-fc2b37a03e7c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57978e83 2021-05-04T01:46:58.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e01a47db-fb63-4954-8a22-fc2b37a03e7c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:46:58.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e01a47db-fb63-4954-8a22-fc2b37a03e7c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:47:00.914Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e01a47db-fb63-4954-8a22-fc2b37a03e7c ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:47:00.914Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=e01a47db-fb63-4954-8a22-fc2b37a03e7c ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:47:01.555Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eec28e2a-66a1-47f8-8737-37ff2376d70d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:47:01.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eec28e2a-66a1-47f8-8737-37ff2376d70d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d0894ee 2021-05-04T01:47:01.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eec28e2a-66a1-47f8-8737-37ff2376d70d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:47:01.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eec28e2a-66a1-47f8-8737-37ff2376d70d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:47:02.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fbe32ea-58d9-45b1-831b-4a4b60e73be9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:47:02.799Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fbe32ea-58d9-45b1-831b-4a4b60e73be9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1fe85764 2021-05-04T01:47:02.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fbe32ea-58d9-45b1-831b-4a4b60e73be9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:47:02.800Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fbe32ea-58d9-45b1-831b-4a4b60e73be9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:47:05.807Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=3fbe32ea-58d9-45b1-831b-4a4b60e73be9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:47:05.807Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=3fbe32ea-58d9-45b1-831b-4a4b60e73be9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:47:05.807Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3fbe32ea-58d9-45b1-831b-4a4b60e73be9 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:47:05.808Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=3fbe32ea-58d9-45b1-831b-4a4b60e73be9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:47:10.816Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:47:10.817Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:47:16.822Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:47:16.823Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:47:16.824Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:47:16.824Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.235.132:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:47:23.829Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:47:23.830Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:47:23.903Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:47:23.903Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.235.132:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:47:25.813Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:47:25.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab1f36d3-86f3-4fc9-b368-847c34ee0676 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:47:25.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab1f36d3-86f3-4fc9-b368-847c34ee0676 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@96ec688 2021-05-04T01:47:25.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab1f36d3-86f3-4fc9-b368-847c34ee0676 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:47:25.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab1f36d3-86f3-4fc9-b368-847c34ee0676 PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:47:27.028Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:47:27.033Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:47:27.899Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:47:27.900Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.95.7:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:47:27.908Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:47:27.910Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.235.132:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:47:28.034Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:47:28.038Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:47:28.875Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=ab1f36d3-86f3-4fc9-b368-847c34ee0676 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:47:28.875Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=ab1f36d3-86f3-4fc9-b368-847c34ee0676 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:47:28.876Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ab1f36d3-86f3-4fc9-b368-847c34ee0676 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:47:28.876Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=ab1f36d3-86f3-4fc9-b368-847c34ee0676 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:47:30.038Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:47:31.288Z [pool-19-thread-1] INFO c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine GET http://message-router.onap:3904/topics (anonymous) ... 2021-05-04T01:47:31.293Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (java.net.SocketException) caught when processing request to {}->http://message-router.onap:3904: Connection reset 2021-05-04T01:47:31.293Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:47:31.295Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:47:31.295Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:47:31.297Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://message-router.onap:3904: The target server failed to respond 2021-05-04T01:47:31.297Z [pool-19-thread-1] INFO o.a.http.impl.execchain.RetryExec RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Retrying request to {}->http://message-router.onap:3904 2021-05-04T01:47:31.298Z [pool-19-thread-1] WARN c.att.nsa.apiClient.http.HttpClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error executing HTTP request. Connection reset; blacklisting for 2 minutes 2021-05-04T01:47:31.299Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=getTopics Connection problem towards U-EB server. Reason: 500 2021-05-04T01:47:31.299Z [pool-19-thread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=getTopics TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=initDistributionEngine Error occured during access to U-EB Server. Operation: try retrieve list of topics from U-EB server 2021-05-04T01:47:34.053Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:48:00.694Z [main] WARN o.o.s.c.util.YamlToObjectConverter RequestID=null ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= The file /var/lib/jetty/config/catalog-be/neo4j-errors-configuration.yaml cannot be found. Ignore reading configuration. 2021-05-04T01:48:00.862Z [main] INFO o.o.s.be.listen.BEAppContextListener RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ASDC version from MANIFEST is 1.8.5 2021-05-04T01:48:00.911Z [main] INFO o.o.s.b.m.BeMonitoringService RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= BE monitoring service is disabled 2021-05-04T01:48:00.911Z [main] INFO o.s.web.context.ContextLoader RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Root WebApplicationContext: initialization started 2021-05-04T01:48:03.106Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphClient created 2021-05-04T01:48:03.110Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** createGraph started ** 2021-05-04T01:48:03.110Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T01:48:03.110Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T01:48:03.484Z [main] INFO com.datastax.driver.core RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= DataStax Java driver 3.8.0 for Apache Cassandra 2021-05-04T01:48:03.489Z [main] INFO c.d.driver.core.GuavaCompatibility RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Detected Guava >= 19 in the classpath, using modern compatibility layer 2021-05-04T01:48:04.112Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T01:48:04.327Z [main] INFO com.datastax.driver.core.NettyUtil RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Found Netty's native epoll transport in the classpath, using it 2021-05-04T01:48:07.533Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:48:07.535Z [main] INFO com.datastax.driver.core.Cluster RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:48:07.535Z [main] INFO com.datastax.driver.core.Cluster RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.95.7:9042 added 2021-05-04T01:48:07.535Z [main] INFO com.datastax.driver.core.Cluster RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:48:11.792Z [main] WARN o.j.g.c.GraphDatabaseConfiguration RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Local setting storage.cql.replication-strategy-options=[Pod, 3] (Type: FIXED) is overridden by globally managed value ([Pod, 3]). Use the ManagementSystem interface instead of the local configuration to control this setting. 2021-05-04T01:48:13.935Z [main] INFO o.j.g.c.GraphDatabaseConfiguration RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Generated unique-instance-id=0af2eb9a251-dev-sdc-be-69f59769db-d8vzh1 2021-05-04T01:48:13.949Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T01:48:25.970Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T01:48:25.972Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Scheduling reconnect task org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask@d4f2f3d with interval of 3 seconds 2021-05-04T01:48:25.974Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T01:48:25.974Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T01:48:25.979Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T01:48:26.012Z [main] INFO o.o.s.b.d.j.JanusGraphGenericDao RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphGenericDao created 2021-05-04T01:48:26.274Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= All sdc keyspaces are : [sdcartifact, sdcrepository, sdcaudit] 2021-05-04T01:48:26.275Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connecting to node: [cassandra.onap] port: 9042. 2021-05-04T01:48:26.276Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connection timeout in seconds : 120 2021-05-04T01:48:26.276Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T01:48:26.276Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T01:48:26.276Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T01:48:26.277Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T01:48:26.277Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** cluster created 2021-05-04T01:48:26.277Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= creating cluster for Cassandra Health Check. 2021-05-04T01:48:26.556Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:48:26.556Z [main] INFO com.datastax.driver.core.Cluster RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:48:26.556Z [main] INFO com.datastax.driver.core.Cluster RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.95.7:9042 added 2021-05-04T01:48:26.556Z [main] INFO com.datastax.driver.core.Cluster RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.235.132:9042 added 2021-05-04T01:48:26.556Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Cluster Metadata: com.datastax.driver.core.Metadata@63541960 2021-05-04T01:48:26.556Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= keyspace : sdcartifact , replication: {Pod=3, class=org.apache.cassandra.locator.NetworkTopologyStrategy} 2021-05-04T01:48:26.557Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= keyspace : sdcrepository , replication: {Pod=3, class=org.apache.cassandra.locator.NetworkTopologyStrategy} 2021-05-04T01:48:26.557Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= keyspace : sdcaudit , replication: {Pod=3, class=org.apache.cassandra.locator.NetworkTopologyStrategy} 2021-05-04T01:48:26.557Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= maxReplication Factor is: 3 2021-05-04T01:48:26.557Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localQuorum is: 2 2021-05-04T01:48:26.557Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Health Check formula : Replication Factor – Local_Quorum = 1 2021-05-04T01:48:26.574Z [main] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ECOMP Portal health check configuration is missing. 2021-05-04T01:48:26.588Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Enter init method of SwitchoverDetector 2021-05-04T01:48:26.588Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null switchover detector service is disabled 2021-05-04T01:48:26.854Z [JanusGraph-Reconnect-Task] INFO c.d.d.c.p.DCAwareRoundRobinPolicy Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:48:26.854Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:48:26.854Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.95.7:9042 added 2021-05-04T01:48:26.854Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:48:26.885Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null creating cluster for Cassandra for monitoring. 2021-05-04T01:48:26.886Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra session is com.datastax.driver.core.SessionManager@699fe19f 2021-05-04T01:48:26.886Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The number of cassandra nodes is:3 2021-05-04T01:48:26.897Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra down nodes number is 0 2021-05-04T01:48:27.602Z [JanusGraph-Reconnect-Task] WARN o.j.g.c.GraphDatabaseConfiguration Local setting storage.cql.replication-strategy-options=[Pod, 3] (Type: FIXED) is overridden by globally managed value ([Pod, 3]). Use the ManagementSystem interface instead of the local configuration to control this setting. 2021-05-04T01:48:28.099Z [main] INFO o.o.s.be.dao.jsongraph.JanusGraphDao RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** JanusGraphDao created 2021-05-04T01:48:28.550Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c4aee1f-39f2-4462-9698-06f82706ea19 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:48:28.550Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c4aee1f-39f2-4462-9698-06f82706ea19 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31329725 2021-05-04T01:48:28.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c4aee1f-39f2-4462-9698-06f82706ea19 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:48:28.551Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c4aee1f-39f2-4462-9698-06f82706ea19 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:48:29.658Z [JanusGraph-Reconnect-Task] INFO o.j.g.c.GraphDatabaseConfiguration Generated unique-instance-id=0af2eb9a251-dev-sdc-be-69f59769db-d8vzh2 2021-05-04T01:48:29.660Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T01:48:30.221Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T01:48:30.221Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T01:48:30.221Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T01:48:30.221Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using native clock to generate timestamps. 2021-05-04T01:48:30.223Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient created 2021-05-04T01:48:31.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd8215f1-90c7-4c39-8adc-e80b4207783d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:48:31.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd8215f1-90c7-4c39-8adc-e80b4207783d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@141ff028 2021-05-04T01:48:31.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd8215f1-90c7-4c39-8adc-e80b4207783d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:48:31.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd8215f1-90c7-4c39-8adc-e80b4207783d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:48:31.986Z [JanusGraph-Reconnect-Task] INFO c.d.d.c.p.DCAwareRoundRobinPolicy Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:48:31.987Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:48:31.987Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.95.7:9042 added 2021-05-04T01:48:31.987Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:48:31.998Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:48:31.998Z [main] INFO com.datastax.driver.core.Cluster RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:48:31.998Z [main] INFO com.datastax.driver.core.Cluster RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.95.7:9042 added 2021-05-04T01:48:31.998Z [main] INFO com.datastax.driver.core.Cluster RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.235.132:9042 added 2021-05-04T01:48:32.762Z [main] INFO o.o.s.b.d.c.AuditCassandraDao RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** AuditCassandraDao created 2021-05-04T01:48:32.977Z [JanusGraph-Reconnect-Task] INFO org.janusgraph.diskstorage.Backend Initiated backend operations thread pool of size 2 2021-05-04T01:48:32.940Z [main] INFO o.o.s.b.c.d.e.DmaapClientFactory RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1620092863, consumerId=sdc-AUTO1-1620092863, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/var/lib/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-05-04T01:48:33.048Z [main] ERROR o.o.s.be.catalog.impl.DmaapProducer RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-05-04T01:48:41.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7478b078-ae6c-47c2-82a1-f51bd033c150 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:48:41.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7478b078-ae6c-47c2-82a1-f51bd033c150 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f3f8426 2021-05-04T01:48:41.479Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7478b078-ae6c-47c2-82a1-f51bd033c150 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:48:41.479Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7478b078-ae6c-47c2-82a1-f51bd033c150 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:48:41.720Z [main] INFO o.o.s.b.d.c.ArtifactCassandraDao RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** ArtifactCassandraDao created 2021-05-04T01:48:41.832Z [main] ERROR o.o.s.b.tosca.PolicyExportParserImpl RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Failed to retrieve all data types NOT_CREATED 2021-05-04T01:48:41.835Z [main] WARN o.s.w.c.s.XmlWebApplicationContext RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'artifactServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/ArtifactServlet.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'artifactBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/components/impl/ArtifactsBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'tosca-export-handler' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/tosca/ToscaExportHandler.class]: Unsatisfied dependency expressed through constructor parameter 3; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'policyExportParserImpl' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/tosca/PolicyExportParserImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.openecomp.sdc.be.tosca.PolicyExportParserImpl]: Constructor threw exception; nested exception is org.openecomp.sdc.be.components.impl.exceptions.SdcResourceNotFoundException 2021-05-04T01:48:43.849Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T01:48:44.524Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7478b078-ae6c-47c2-82a1-f51bd033c150 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:48:44.527Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7478b078-ae6c-47c2-82a1-f51bd033c150 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:48:44.528Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7478b078-ae6c-47c2-82a1-f51bd033c150 ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:48:44.528Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=7478b078-ae6c-47c2-82a1-f51bd033c150 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:48:45.864Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T01:48:45.864Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** sdcSchemaUtils cluster closed 2021-05-04T01:48:45.865Z [main] WARN o.s.c.a.CommonAnnotationBeanPostProcessor RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Destroy method on bean with name 'janusgraph-client' threw an exception: java.lang.NullPointerException 2021-05-04T01:48:45.880Z [main] ERROR o.s.web.context.ContextLoader RequestID=5db31305-4d0c-4f76-bbc0-96af67afe823 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Context initialization failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'artifactServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/ArtifactServlet.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'artifactBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/components/impl/ArtifactsBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'tosca-export-handler' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/tosca/ToscaExportHandler.class]: Unsatisfied dependency expressed through constructor parameter 3; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'policyExportParserImpl' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/tosca/PolicyExportParserImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.openecomp.sdc.be.tosca.PolicyExportParserImpl]: Constructor threw exception; nested exception is org.openecomp.sdc.be.components.impl.exceptions.SdcResourceNotFoundException at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'artifactBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/components/impl/ArtifactsBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'tosca-export-handler' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/tosca/ToscaExportHandler.class]: Unsatisfied dependency expressed through constructor parameter 3; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'policyExportParserImpl' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/tosca/PolicyExportParserImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.openecomp.sdc.be.tosca.PolicyExportParserImpl]: Constructor threw exception; nested exception is org.openecomp.sdc.be.components.impl.exceptions.SdcResourceNotFoundException at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 59 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'tosca-export-handler' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/tosca/ToscaExportHandler.class]: Unsatisfied dependency expressed through constructor parameter 3; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'policyExportParserImpl' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/tosca/PolicyExportParserImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.openecomp.sdc.be.tosca.PolicyExportParserImpl]: Constructor threw exception; nested exception is org.openecomp.sdc.be.components.impl.exceptions.SdcResourceNotFoundException at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 73 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'policyExportParserImpl' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-4599297242770365753.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/tosca/PolicyExportParserImpl.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.openecomp.sdc.be.tosca.PolicyExportParserImpl]: Constructor threw exception; nested exception is org.openecomp.sdc.be.components.impl.exceptions.SdcResourceNotFoundException at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:313) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:294) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 87 common frames omitted Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.openecomp.sdc.be.tosca.PolicyExportParserImpl]: Constructor threw exception; nested exception is org.openecomp.sdc.be.components.impl.exceptions.SdcResourceNotFoundException at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:217) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:309) ... 101 common frames omitted Caused by: org.openecomp.sdc.be.components.impl.exceptions.SdcResourceNotFoundException: null at org.openecomp.sdc.be.tosca.PolicyExportParserImpl.getDataTypes(PolicyExportParserImpl.java:74) at org.openecomp.sdc.be.tosca.PolicyExportParserImpl.(PolicyExportParserImpl.java:67) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:204) ... 103 common frames omitted 2021-05-04T01:49:11.251Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T01:49:11.251Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T01:49:11.251Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T01:49:11.256Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T01:49:18.191Z [JanusGraph-Reconnect-Task] INFO c.d.d.c.p.DCAwareRoundRobinPolicy Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:49:18.191Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:49:18.191Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.95.7:9042 added 2021-05-04T01:49:18.191Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:49:22.657Z [JanusGraph-Reconnect-Task] WARN o.j.g.c.GraphDatabaseConfiguration Local setting storage.cql.replication-strategy-options=[Pod, 3] (Type: FIXED) is overridden by globally managed value ([Pod, 3]). Use the ManagementSystem interface instead of the local configuration to control this setting. 2021-05-04T01:49:24.748Z [JanusGraph-Reconnect-Task] INFO o.j.g.c.GraphDatabaseConfiguration Generated unique-instance-id=0af2eb9a251-dev-sdc-be-69f59769db-d8vzh3 2021-05-04T01:49:24.753Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T01:49:31.597Z [JanusGraph-Reconnect-Task] INFO c.d.d.c.p.DCAwareRoundRobinPolicy Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:49:31.597Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:49:31.597Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.95.7:9042 added 2021-05-04T01:49:31.597Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:49:49.604Z [JanusGraph-Reconnect-Task] INFO org.janusgraph.diskstorage.Backend Initiated backend operations thread pool of size 2 2021-05-04T01:49:53.092Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:49:54.101Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:49:56.117Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:00.123Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:08.129Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:13.588Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:14.608Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:24.152Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:28.698Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:32.705Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:36.679Z [JanusGraph Cluster-worker-0] ERROR c.d.driver.core.RequestHandler Unexpected error while querying cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool is shutdown at com.datastax.driver.core.HostConnectionPool.closeAsync(HostConnectionPool.java:660) at com.datastax.driver.core.SessionManager.removePool(SessionManager.java:436) at com.datastax.driver.core.SessionManager.onDown(SessionManager.java:524) at com.datastax.driver.core.Cluster$Manager.onDown(Cluster.java:2173) at com.datastax.driver.core.Cluster$Manager.access$1200(Cluster.java:1556) at com.datastax.driver.core.Cluster$Manager$5.runMayThrow(Cluster.java:2128) at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:50:36.888Z [JanusGraph-Reconnect-Task] INFO o.j.diskstorage.log.kcvs.KCVSLog Loaded unidentified ReadMarker start time 2021-05-04T01:50:36.888592Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@408aca0d 2021-05-04T01:50:37.064Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** JanusGraph graph created 2021-05-04T01:50:37.064Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** Cancelling JanusGraph reconnect task 2021-05-04T01:50:40.711Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:56.203Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:50:56.719Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:51:17.114Z [pool-19-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:51:06.524696Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T01:51:28.727Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T01:56:00.085Z [main] WARN o.o.s.c.util.YamlToObjectConverter RequestID=null ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= The file /var/lib/jetty/config/catalog-be/neo4j-errors-configuration.yaml cannot be found. Ignore reading configuration. 2021-05-04T01:56:00.216Z [main] INFO o.o.s.be.listen.BEAppContextListener RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ASDC version from MANIFEST is 1.8.5 2021-05-04T01:56:00.261Z [main] INFO o.o.s.b.m.BeMonitoringService RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= BE monitoring service is disabled 2021-05-04T01:56:00.262Z [main] INFO o.s.web.context.ContextLoader RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Root WebApplicationContext: initialization started 2021-05-04T01:56:02.422Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphClient created 2021-05-04T01:56:02.426Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** createGraph started ** 2021-05-04T01:56:02.427Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T01:56:02.427Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T01:56:02.787Z [main] INFO com.datastax.driver.core RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= DataStax Java driver 3.8.0 for Apache Cassandra 2021-05-04T01:56:02.792Z [main] INFO c.d.driver.core.GuavaCompatibility RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Detected Guava >= 19 in the classpath, using modern compatibility layer 2021-05-04T01:56:03.347Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T01:56:03.513Z [main] INFO com.datastax.driver.core.NettyUtil RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Found Netty's native epoll transport in the classpath, using it 2021-05-04T01:56:04.435Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:56:04.436Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.8.171:9042 added 2021-05-04T01:56:04.437Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.95.7:9042 added 2021-05-04T01:56:04.437Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:56:06.636Z [main] WARN o.j.g.c.GraphDatabaseConfiguration RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Local setting storage.cql.replication-strategy-options=[Pod, 3] (Type: FIXED) is overridden by globally managed value ([Pod, 3]). Use the ManagementSystem interface instead of the local configuration to control this setting. 2021-05-04T01:56:09.792Z [main] INFO o.j.g.c.GraphDatabaseConfiguration RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Generated unique-instance-id=0af2eb9a252-dev-sdc-be-69f59769db-d8vzh1 2021-05-04T01:56:09.812Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T01:56:10.372Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:56:10.373Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.8.171:9042 added 2021-05-04T01:56:10.373Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.95.7:9042 added 2021-05-04T01:56:10.373Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host /10.242.235.132:9042 added 2021-05-04T01:56:11.392Z [main] INFO org.janusgraph.diskstorage.Backend RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Initiated backend operations thread pool of size 2 2021-05-04T01:56:14.215Z [main] INFO o.j.diskstorage.log.kcvs.KCVSLog RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Loaded unidentified ReadMarker start time 2021-05-04T01:56:14.215321Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@1738d3b2 2021-05-04T01:56:14.457Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraph graph created 2021-05-04T01:56:14.576Z [main] INFO o.o.s.b.d.j.JanusGraphGenericDao RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphGenericDao created 2021-05-04T01:56:14.727Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-05-04T01:56:14.872Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= All sdc keyspaces are : [sdcartifact, sdcrepository, sdcaudit] 2021-05-04T01:56:14.873Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connecting to node: [cassandra.onap] port: 9042. 2021-05-04T01:56:14.874Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connection timeout in seconds : 120 2021-05-04T01:56:14.874Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T01:56:14.874Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T01:56:14.875Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T01:56:14.875Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T01:56:14.875Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** cluster created 2021-05-04T01:56:14.875Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= creating cluster for Cassandra Health Check. 2021-05-04T01:56:15.234Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:56:15.234Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:56:15.234Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.95.7:9042 added 2021-05-04T01:56:15.234Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= New Cassandra host cassandra.onap/10.242.235.132:9042 added 2021-05-04T01:56:15.234Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Cluster Metadata: com.datastax.driver.core.Metadata@1e6be9f0 2021-05-04T01:56:15.235Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= keyspace : sdcartifact , replication: {Pod=3, class=org.apache.cassandra.locator.NetworkTopologyStrategy} 2021-05-04T01:56:15.235Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= keyspace : sdcrepository , replication: {Pod=3, class=org.apache.cassandra.locator.NetworkTopologyStrategy} 2021-05-04T01:56:15.235Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= keyspace : sdcaudit , replication: {Pod=3, class=org.apache.cassandra.locator.NetworkTopologyStrategy} 2021-05-04T01:56:15.235Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= maxReplication Factor is: 3 2021-05-04T01:56:15.235Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localQuorum is: 2 2021-05-04T01:56:15.235Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Health Check formula : Replication Factor – Local_Quorum = 1 2021-05-04T01:56:15.253Z [main] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ECOMP Portal health check configuration is missing. 2021-05-04T01:56:15.266Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Enter init method of SwitchoverDetector 2021-05-04T01:56:15.266Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null switchover detector service is disabled 2021-05-04T01:56:17.382Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null creating cluster for Cassandra for monitoring. 2021-05-04T01:56:17.382Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra session is com.datastax.driver.core.SessionManager@2e8417ea 2021-05-04T01:56:17.383Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The number of cassandra nodes is:3 2021-05-04T01:56:17.386Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra down nodes number is 0 2021-05-04T01:56:18.712Z [main] INFO o.o.s.be.dao.jsongraph.JanusGraphDao RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** JanusGraphDao created 2021-05-04T01:56:21.008Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T01:56:21.009Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T01:56:21.009Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T01:56:21.009Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using native clock to generate timestamps. 2021-05-04T01:56:21.010Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient created 2021-05-04T01:56:21.450Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c668c089-5919-4d1f-8bf4-cc1cb6bb93de PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:56:21.450Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c668c089-5919-4d1f-8bf4-cc1cb6bb93de PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ab397a7 2021-05-04T01:56:21.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c668c089-5919-4d1f-8bf4-cc1cb6bb93de PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:56:21.451Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c668c089-5919-4d1f-8bf4-cc1cb6bb93de PartnerName=SDC.BE The cassandra down nodes number is 0 2021-05-04T01:56:22.056Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T01:56:22.061Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T01:56:22.061Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.95.7:9042 added 2021-05-04T01:56:22.061Z [main] INFO com.datastax.driver.core.Cluster RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.235.132:9042 added 2021-05-04T01:56:24.474Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c668c089-5919-4d1f-8bf4-cc1cb6bb93de ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:56:24.480Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c668c089-5919-4d1f-8bf4-cc1cb6bb93de ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:56:24.488Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c668c089-5919-4d1f-8bf4-cc1cb6bb93de ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:56:24.488Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c668c089-5919-4d1f-8bf4-cc1cb6bb93de ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:56:27.527Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:56:27.529Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-05-04T01:56:29.523Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:56:29.697Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:56:30.527Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:56:33.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:56:35.541Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:56:35.543Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.235.132:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:56:36.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:56:39.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:56:42.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:56:42.600Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-05-04T01:56:42.601Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.235.132:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:56:43.732Z [main] WARN o.s.w.c.s.XmlWebApplicationContext RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Error preparing queries for accessor AuditAccessor 2021-05-04T01:56:44.594Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:56:44.596Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.8.171:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:56:44.602Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:56:44.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5573ed53-38c0-41f4-a9cf-80c1f3d3970a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T01:56:44.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5573ed53-38c0-41f4-a9cf-80c1f3d3970a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16b13e0 2021-05-04T01:56:44.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5573ed53-38c0-41f4-a9cf-80c1f3d3970a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-05-04T01:56:44.603Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5573ed53-38c0-41f4-a9cf-80c1f3d3970a PartnerName=SDC.BE The cassandra down nodes number is 2 2021-05-04T01:56:45.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:56:45.771Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T01:56:47.704Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5573ed53-38c0-41f4-a9cf-80c1f3d3970a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-05-04T01:56:47.706Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5573ed53-38c0-41f4-a9cf-80c1f3d3970a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted 2021-05-04T01:56:47.707Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5573ed53-38c0-41f4-a9cf-80c1f3d3970a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component CASSANDRA state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-05-04T01:56:47.708Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=5573ed53-38c0-41f4-a9cf-80c1f3d3970a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-05-04T01:56:47.801Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T01:56:47.801Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** sdcSchemaUtils cluster closed 2021-05-04T01:56:48.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-05-04T01:56:48.819Z [main] ERROR o.j.diskstorage.log.kcvs.KCVSLog RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Reader thread pool for KCVSLog systemlog did not shut down in time - could not clean up or set read markers 2021-05-04T01:56:48.823Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-05-04T01:56:23.827922Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent failure in storage backend at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.interruptibleWait(CQLKeyColumnValueStore.java:313) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:289) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: java.lang.InterruptedException: null at java.base/java.lang.Object.wait(Native Method) at java.base/java.lang.Object.wait(Unknown Source) at io.vavr.control.Try.run(Try.java:105) at io.vavr.concurrent.FutureImpl.await(FutureImpl.java:114) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.interruptibleWait(CQLKeyColumnValueStore.java:308) ... 15 common frames omitted 2021-05-04T01:56:50.737Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:56:53.531Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:56:56.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:56:59.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:02.527Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:05.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:07.763Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** JanusGraphClient session closed 2021-05-04T01:57:07.776Z [main] ERROR o.s.web.context.ContextLoader RequestID=dfea44c3-0f74-474a-bf22-781b21dc8779 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Context initialization failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Error preparing queries for accessor AuditAccessor at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Error preparing queries for accessor AuditAccessor at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 59 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Error preparing queries for accessor AuditAccessor at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 73 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8389293098567192595.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Error preparing queries for accessor AuditAccessor at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 87 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Error preparing queries for accessor AuditAccessor at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:415) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 101 common frames omitted Caused by: java.lang.RuntimeException: Error preparing queries for accessor AuditAccessor at com.datastax.driver.mapping.AccessorMapper.prepare(AccessorMapper.java:63) at com.datastax.driver.mapping.MappingManager.getAccessor(MappingManager.java:359) at com.datastax.driver.mapping.MappingManager.createAccessor(MappingManager.java:321) at org.openecomp.sdc.be.dao.cassandra.AuditCassandraDao.init(AuditCassandraDao.java:60) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ... 114 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Timed out waiting for server response at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:104) at com.datastax.driver.mapping.AccessorMapper.prepare(AccessorMapper.java:58) ... 124 common frames omitted Caused by: com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Timed out waiting for server response at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onTimeout(RequestHandler.java:973) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T01:57:08.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:11.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:14.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:17.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:20.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:23.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:26.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:29.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:32.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:35.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:38.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:41.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:44.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:47.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:50.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:53.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:56.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:57:59.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:02.534Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:05.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:08.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:11.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:14.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:17.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:20.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:23.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:26.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:29.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:32.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:35.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:38.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:41.527Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:44.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:47.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:50.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:53.527Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:56.529Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:58:59.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:02.537Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:05.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:08.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:11.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:14.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:17.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:20.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:23.527Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:26.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:29.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:32.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:35.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:38.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:41.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:44.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:47.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:50.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:53.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:56.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T01:59:59.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:02.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:05.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:08.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:11.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:14.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:17.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:20.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:23.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:26.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:29.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:32.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:35.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:38.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:41.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:44.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:47.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:50.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:53.530Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:56.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:00:59.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:02.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:05.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:08.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:11.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:14.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:17.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:20.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:23.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:26.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:29.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:32.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:35.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:38.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:41.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:44.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:47.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:50.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:53.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:56.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:01:59.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:02.534Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:05.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:08.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:11.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:14.529Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:17.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:20.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:23.527Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:26.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:29.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:32.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:35.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:38.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:41.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:44.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:47.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:50.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:53.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:56.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:02:59.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:02.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:05.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:08.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:11.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:14.528Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:17.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:20.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:23.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:26.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:29.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:32.525Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:35.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:03:38.526Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.lang.IllegalStateException: Graph has been closed 2021-05-04T02:04:00.755Z [main] WARN o.o.s.c.util.YamlToObjectConverter RequestID=null ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= The file /var/lib/jetty/config/catalog-be/neo4j-errors-configuration.yaml cannot be found. Ignore reading configuration. 2021-05-04T02:04:00.894Z [main] INFO o.o.s.be.listen.BEAppContextListener RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ASDC version from MANIFEST is 1.8.5 2021-05-04T02:04:00.944Z [main] INFO o.o.s.b.m.BeMonitoringService RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= BE monitoring service is disabled 2021-05-04T02:04:00.945Z [main] INFO o.s.web.context.ContextLoader RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Root WebApplicationContext: initialization started 2021-05-04T02:04:03.206Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphClient created 2021-05-04T02:04:03.215Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** createGraph started ** 2021-05-04T02:04:03.215Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:03.215Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:03.587Z [main] INFO com.datastax.driver.core RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= DataStax Java driver 3.8.0 for Apache Cassandra 2021-05-04T02:04:03.592Z [main] INFO c.d.driver.core.GuavaCompatibility RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Detected Guava >= 19 in the classpath, using modern compatibility layer 2021-05-04T02:04:04.250Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T02:04:04.437Z [main] INFO com.datastax.driver.core.NettyUtil RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Found Netty's native epoll transport in the classpath, using it 2021-05-04T02:04:05.030Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:05.032Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Scheduling reconnect task org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask@2688b2be with interval of 3 seconds 2021-05-04T02:04:05.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:05.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:05.038Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:05.082Z [main] INFO o.o.s.b.d.j.JanusGraphGenericDao RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphGenericDao created 2021-05-04T02:04:05.120Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:05.405Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= All sdc keyspaces are : [sdcartifact, sdcrepository, sdcaudit] 2021-05-04T02:04:05.405Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connecting to node: [cassandra.onap] port: 9042. 2021-05-04T02:04:05.407Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connection timeout in seconds : 120 2021-05-04T02:04:05.408Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T02:04:05.408Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T02:04:05.408Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T02:04:05.408Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T02:04:05.408Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** cluster created 2021-05-04T02:04:05.408Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= creating cluster for Cassandra Health Check. 2021-05-04T02:04:05.439Z [main] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null create cassandra cluster failed with exception.: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:04:05.467Z [main] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ECOMP Portal health check configuration is missing. 2021-05-04T02:04:05.488Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Enter init method of SwitchoverDetector 2021-05-04T02:04:05.489Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null switchover detector service is disabled 2021-05-04T02:04:05.498Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null creating cluster for Cassandra for monitoring. 2021-05-04T02:04:05.499Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra session is null 2021-05-04T02:04:05.499Z [main] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:06.035Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=390ce61c-29e3-4bb7-99b2-b1570503121a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:06.035Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=390ce61c-29e3-4bb7-99b2-b1570503121a PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:06.036Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=390ce61c-29e3-4bb7-99b2-b1570503121a ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:06.143Z [main] INFO o.o.s.be.dao.jsongraph.JanusGraphDao RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** JanusGraphDao created 2021-05-04T02:04:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:08.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:08.065Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:08.358Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T02:04:08.358Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T02:04:08.358Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T02:04:08.358Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using native clock to generate timestamps. 2021-05-04T02:04:08.359Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient created 2021-05-04T02:04:09.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f24e1434-5677-4912-99f1-cf4ce0a433ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:09.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f24e1434-5677-4912-99f1-cf4ce0a433ba PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:09.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=f24e1434-5677-4912-99f1-cf4ce0a433ba ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:11.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:11.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:11.037Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:11.052Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:12.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e1e2e41-5c53-47b4-aacf-8167e3ee8b4d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:12.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e1e2e41-5c53-47b4-aacf-8167e3ee8b4d PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:12.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e1e2e41-5c53-47b4-aacf-8167e3ee8b4d ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:14.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:14.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:14.037Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:14.106Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:15.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=268635ab-4600-4442-b266-da0fbb976625 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:15.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=268635ab-4600-4442-b266-da0fbb976625 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:15.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=268635ab-4600-4442-b266-da0fbb976625 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:17.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:17.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:17.037Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:17.053Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:18.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b4202b1-c7cd-4461-9a4b-d83773038d6d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:18.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b4202b1-c7cd-4461-9a4b-d83773038d6d PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:18.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b4202b1-c7cd-4461-9a4b-d83773038d6d ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:20.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:20.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:20.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:20.050Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:21.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1f9ab5a-5c04-4b2f-a804-fd9559170585 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:21.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1f9ab5a-5c04-4b2f-a804-fd9559170585 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:21.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1f9ab5a-5c04-4b2f-a804-fd9559170585 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:23.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:23.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:23.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:23.103Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:24.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=719b1895-a137-4471-8f3a-eadbbba3cc1e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:24.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=719b1895-a137-4471-8f3a-eadbbba3cc1e PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:24.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=719b1895-a137-4471-8f3a-eadbbba3cc1e ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:26.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:26.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:26.037Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:26.053Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:27.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a91cc09-9ebf-46c4-8dd3-ac208d7fdc2f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:27.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a91cc09-9ebf-46c4-8dd3-ac208d7fdc2f PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:27.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a91cc09-9ebf-46c4-8dd3-ac208d7fdc2f ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:29.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:29.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:29.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:29.049Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:30.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81ec1e8e-a8f8-4ff1-9da1-5bd629866e3d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:30.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81ec1e8e-a8f8-4ff1-9da1-5bd629866e3d PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:30.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=81ec1e8e-a8f8-4ff1-9da1-5bd629866e3d ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:32.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:32.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:32.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:32.049Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:33.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a168ec-fabd-4775-9650-e00a4e23b8f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:33.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a168ec-fabd-4775-9650-e00a4e23b8f8 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:33.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a168ec-fabd-4775-9650-e00a4e23b8f8 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:33.047Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=81a168ec-fabd-4775-9650-e00a4e23b8f8 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://sdc-onboarding-be.onap:8445: The target server failed to respond 2021-05-04T02:04:33.048Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=81a168ec-fabd-4775-9650-e00a4e23b8f8 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Retrying request to {s}->https://sdc-onboarding-be.onap:8445 2021-05-04T02:04:33.077Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=81a168ec-fabd-4775-9650-e00a4e23b8f8 ErrorCategory=ERROR ServiceName=SDC catalog ServerIPAddress=10.242.235.154 TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T02:04:33.080Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=81a168ec-fabd-4775-9650-e00a4e23b8f8 ErrorCategory=ERROR ServiceName=SDC catalog ServerIPAddress=10.242.235.154 TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T02:04:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:35.040Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:35.060Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:36.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1b51490-814d-4683-bae6-ce518523200e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:36.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1b51490-814d-4683-bae6-ce518523200e PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:36.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=d1b51490-814d-4683-bae6-ce518523200e ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:36.039Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d1b51490-814d-4683-bae6-ce518523200e ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) 2021-05-04T02:04:36.040Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d1b51490-814d-4683-bae6-ce518523200e ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T02:04:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:38.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:38.102Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:39.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=265e7c68-a5d4-45e5-8373-8975b97debef PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:39.034Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=265e7c68-a5d4-45e5-8373-8975b97debef PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:39.034Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=265e7c68-a5d4-45e5-8373-8975b97debef ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:41.036Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:41.036Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:41.040Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:41.065Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:41.595Z [main] INFO o.o.s.b.d.c.AuditCassandraDao RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** AuditCassandraDao failed 2021-05-04T02:04:41.610Z [main] WARN o.s.w.c.s.XmlWebApplicationContext RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED 2021-05-04T02:04:42.058Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=265e7c68-a5d4-45e5-8373-8975b97debef ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T02:04:42.059Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=265e7c68-a5d4-45e5-8373-8975b97debef ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T02:04:42.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abeae322-82db-4450-8a51-8146ec61278f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:04:42.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abeae322-82db-4450-8a51-8146ec61278f PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:04:42.059Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=abeae322-82db-4450-8a51-8146ec61278f ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:04:43.601Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T02:04:43.602Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T02:04:43.602Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** sdcSchemaUtils cluster closed 2021-05-04T02:04:43.604Z [main] WARN o.s.c.a.CommonAnnotationBeanPostProcessor RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Destroy method on bean with name 'janusgraph-client' threw an exception: java.lang.NullPointerException 2021-05-04T02:04:43.613Z [main] ERROR o.s.web.context.ContextLoader RequestID=c40e1bed-b621-4168-8c6c-972229b7141e ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Context initialization failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 59 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 73 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-9218273918840410009.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 87 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:415) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 101 common frames omitted Caused by: java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.openecomp.sdc.be.dao.cassandra.AuditCassandraDao.init(AuditCassandraDao.java:65) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ... 114 common frames omitted 2021-05-04T02:04:44.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:44.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:44.040Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:44.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:45.071Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=abeae322-82db-4450-8a51-8146ec61278f ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out 2021-05-04T02:04:45.073Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=abeae322-82db-4450-8a51-8146ec61278f ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:119) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:61) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:81) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:55) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:258) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:163) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:446) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.19.65] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:111) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-05-04T02:04:47.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:47.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:47.037Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:47.053Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:50.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:50.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:50.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:50.100Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:53.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:53.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:53.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:53.046Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:56.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:56.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:56.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:56.050Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:59.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:04:59.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:04:59.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:04:59.101Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:05:02.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:05:02.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:05:02.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:05:02.049Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:05:05.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:05:05.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:05:05.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:05:05.049Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:05:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:05:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:05:08.038Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:05:19.447Z [JanusGraph-Reconnect-Task] INFO c.d.d.c.p.DCAwareRoundRobinPolicy Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T02:05:19.448Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T02:05:19.448Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.95.7:9042 added 2021-05-04T02:05:19.448Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.235.132:9042 added 2021-05-04T02:05:19.476Z [JanusGraph Cluster-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to /10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:19.498Z [JanusGraph Cluster-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to /10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:20.486Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:20.487Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:20.508Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:20.509Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:22.504Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:22.506Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:22.517Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:22.518Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:26.513Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:26.514Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:26.527Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:26.527Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:33.833Z [JanusGraph-Reconnect-Task] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [getConfiguration]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:98) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:105) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1274) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:78) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.createGraph(JanusGraphClient.java:226) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask.run(JanusGraphClient.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 0 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 0 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 0 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T02:05:34.523Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:34.524Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 16000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:34.546Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:34.547Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 16000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:45.955Z [JanusGraph-Reconnect-Task] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [getConfiguration]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:98) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:105) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1274) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:78) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.createGraph(JanusGraphClient.java:226) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask.run(JanusGraphClient.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra.onap/10.242.8.171:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Timed out waiting for server response)) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra.onap/10.242.8.171:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Timed out waiting for server response)) at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:283) at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:61) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:375) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:557) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:539) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onTimeout(RequestHandler.java:981) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted 2021-05-04T02:05:50.533Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:50.534Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 32000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:50.601Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:05:50.602Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 32000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:05:52.910Z [JanusGraph-Reconnect-Task] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [getConfiguration]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:98) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:105) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1274) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:78) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.createGraph(JanusGraphClient.java:226) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask.run(JanusGraphClient.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T02:05:57.963Z [JanusGraph-Reconnect-Task] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [getConfiguration]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:98) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:105) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1274) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:78) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.createGraph(JanusGraphClient.java:226) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask.run(JanusGraphClient.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T02:06:06.349Z [JanusGraph-Reconnect-Task] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [getConfiguration]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:98) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:105) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1274) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:78) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.createGraph(JanusGraphClient.java:226) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask.run(JanusGraphClient.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T02:06:18.255Z [JanusGraph-Reconnect-Task] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [getConfiguration]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:98) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:105) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1274) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:78) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.createGraph(JanusGraphClient.java:226) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask.run(JanusGraphClient.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T02:06:22.609Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:06:22.610Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 64000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:06:22.615Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:06:22.616Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 64000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:06:23.454Z [JanusGraph-Reconnect-Task] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [getConfiguration]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:98) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:95) at org.janusgraph.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:105) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1274) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:78) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.createGraph(JanusGraphClient.java:226) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask.run(JanusGraphClient.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-05-04T02:06:32.989Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:32.989Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:32.989Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:32.998Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.027Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.027Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.027Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.029Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.042Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.042Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.042Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.043Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.054Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.054Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.054Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.057Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.069Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.069Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.069Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.071Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.085Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.086Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.086Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.087Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.100Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.101Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.101Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.102Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.120Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.120Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.121Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.122Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.142Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.142Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.142Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.144Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.156Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.156Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.156Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.157Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.169Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.169Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.169Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.170Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.180Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.180Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.180Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.181Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.197Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.197Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.198Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.199Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.216Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.216Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.216Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.217Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.242Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.242Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.243Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.244Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.254Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.254Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.254Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.256Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.266Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.266Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.266Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.268Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.276Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.276Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.276Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.277Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.287Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.287Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.287Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.288Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.297Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.297Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.297Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.298Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.307Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.307Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.307Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.308Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.316Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.316Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.316Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.318Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.328Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.338Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.338Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.338Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.339Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.348Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.348Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.348Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.349Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.361Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.361Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.361Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.362Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.371Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.371Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.371Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.372Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.392Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.393Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:33.393Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:33.394Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:33.402Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:35.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:35.070Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:38.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:38.104Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:41.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:41.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:41.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:41.053Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:44.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:44.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:44.044Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:44.056Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:47.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:47.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:47.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:47.047Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:50.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:50.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:50.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:50.102Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:53.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:53.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:53.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:53.048Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:56.034Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:56.034Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:56.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:56.046Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:59.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:06:59.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:06:59.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:06:59.047Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:02.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:02.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:02.044Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:02.061Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:05.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:05.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:05.039Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:05.053Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:08.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:08.103Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:11.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:11.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:11.038Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:11.102Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:14.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:14.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:14.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:14.050Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:17.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:17.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:17.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:17.049Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:20.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:20.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:20.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:20.101Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:23.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:23.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:23.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:23.048Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:26.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:26.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:26.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:26.051Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:29.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:29.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:29.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:29.099Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:32.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:32.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:32.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:32.046Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:35.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:35.049Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:38.037Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:38.049Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:41.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:41.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:41.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:41.051Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:44.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:44.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:44.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:44.105Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:47.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:47.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:47.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:47.046Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:50.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:50.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:50.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:50.047Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:53.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:53.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:53.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:53.105Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:56.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:56.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:56.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:56.102Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:59.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:07:59.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:07:59.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:07:59.044Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:02.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:08:02.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:02.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:08:02.103Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:05.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:08:05.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:05.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:08:05.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:08:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:08.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:08:08.047Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:11.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:08:11.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:11.037Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:08:25.848Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:25.848Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:08:25.848Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:25.852Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:08:37.970Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:37.970Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:08:37.970Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:37.976Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:08:58.491Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:58.491Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:08:58.491Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:08:58.498Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:10.608Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:10.608Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:10.608Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:10.610Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:33.806Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:33.807Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:33.807Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:33.814Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:33.905Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:33.905Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:33.906Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:33.909Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:33.919Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:33.919Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:33.920Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:33.922Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.003Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.003Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.003Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.006Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.218Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.218Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.219Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.221Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.303Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.304Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.304Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.306Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.324Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.324Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.324Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.326Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.403Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.403Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.403Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.405Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.414Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.414Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.414Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.416Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.503Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.503Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.503Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.505Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.514Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.514Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.514Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.516Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.526Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.526Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.526Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.528Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.620Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.620Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.620Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.621Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.633Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.633Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.633Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.635Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.703Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.704Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.704Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.705Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.718Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.718Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.718Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.720Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.733Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.733Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.733Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.734Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.803Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.804Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.804Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.805Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.817Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.817Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.817Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.819Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.829Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.829Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.829Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.830Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.903Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.904Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.904Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.906Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:34.918Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.918Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:34.918Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:34.920Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:35.003Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:35.003Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:35.003Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:35.005Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:35.016Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:35.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:35.044Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:38.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:38.046Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:41.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:41.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:41.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:41.049Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:44.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:44.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:44.037Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:44.051Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:47.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:47.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:47.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:47.049Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:50.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:50.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:50.036Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:50.048Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:53.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:53.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:53.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:53.103Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:56.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:56.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:56.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:56.102Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:59.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:09:59.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:09:59.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:09:59.046Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:02.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:02.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:02.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:02.106Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:05.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:05.033Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:05.039Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:05.103Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:08.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:08.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:08.045Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:11.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:11.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:11.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:11.046Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:14.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:14.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:14.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:14.102Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:17.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:17.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:17.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:17.041Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:20.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:20.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:20.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:20.046Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:23.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:23.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:23.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:23.101Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:26.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:26.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:26.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:26.047Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:29.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:29.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:29.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:29.044Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:32.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:32.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:32.034Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:32.044Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:35.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:35.035Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:10:35.104Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:10:38.032Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:10:38.037Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:11:06.751Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:11:06.752Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:11:06.752Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:11:06.759Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:11:19.953Z [JanusGraph-Reconnect-Task] INFO c.d.d.c.p.DCAwareRoundRobinPolicy Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T02:11:19.954Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T02:11:19.954Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.95.7:9042 added 2021-05-04T02:11:19.954Z [JanusGraph-Reconnect-Task] INFO com.datastax.driver.core.Cluster New Cassandra host /10.242.235.132:9042 added 2021-05-04T02:11:19.964Z [JanusGraph Cluster-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to /10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:11:20.003Z [JanusGraph Cluster-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to /10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:11:20.968Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:11:20.969Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:11:21.018Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:11:21.019Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:11:22.974Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:11:22.976Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:11:23.025Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:11:23.026Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:11:26.983Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:11:26.984Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:11:27.035Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:11:27.036Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:11:35.001Z [JanusGraph Cluster-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:11:35.002Z [JanusGraph Cluster-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.235.132:9042, scheduling retry in 16000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:11:35.042Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:11:35.045Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.95.7:9042, scheduling retry in 16000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:12:00.499Z [main] WARN o.o.s.c.util.YamlToObjectConverter RequestID=null ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= The file /var/lib/jetty/config/catalog-be/neo4j-errors-configuration.yaml cannot be found. Ignore reading configuration. 2021-05-04T02:12:00.672Z [main] INFO o.o.s.be.listen.BEAppContextListener RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ASDC version from MANIFEST is 1.8.5 2021-05-04T02:12:00.721Z [main] INFO o.o.s.b.m.BeMonitoringService RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= BE monitoring service is disabled 2021-05-04T02:12:00.722Z [main] INFO o.s.web.context.ContextLoader RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Root WebApplicationContext: initialization started 2021-05-04T02:12:02.875Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphClient created 2021-05-04T02:12:02.881Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** createGraph started ** 2021-05-04T02:12:02.881Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:02.881Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:03.336Z [main] INFO com.datastax.driver.core RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= DataStax Java driver 3.8.0 for Apache Cassandra 2021-05-04T02:12:03.341Z [main] INFO c.d.driver.core.GuavaCompatibility RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Detected Guava >= 19 in the classpath, using modern compatibility layer 2021-05-04T02:12:03.921Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T02:12:04.118Z [main] INFO com.datastax.driver.core.NettyUtil RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Found Netty's native epoll transport in the classpath, using it 2021-05-04T02:12:27.105Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:27.109Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Scheduling reconnect task org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask@104e6540 with interval of 3 seconds 2021-05-04T02:12:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:27.111Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:27.119Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:27.172Z [main] INFO o.o.s.b.d.j.JanusGraphGenericDao RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphGenericDao created 2021-05-04T02:12:27.473Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= All sdc keyspaces are : [sdcartifact, sdcrepository, sdcaudit] 2021-05-04T02:12:27.474Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connecting to node: [cassandra.onap] port: 9042. 2021-05-04T02:12:27.475Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connection timeout in seconds : 120 2021-05-04T02:12:27.475Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T02:12:27.475Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T02:12:27.475Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T02:12:27.475Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T02:12:27.476Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** cluster created 2021-05-04T02:12:27.476Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= creating cluster for Cassandra Health Check. 2021-05-04T02:12:27.499Z [main] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null create cassandra cluster failed with exception.: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:12:27.521Z [main] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ECOMP Portal health check configuration is missing. 2021-05-04T02:12:27.535Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Enter init method of SwitchoverDetector 2021-05-04T02:12:27.535Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null switchover detector service is disabled 2021-05-04T02:12:27.541Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null creating cluster for Cassandra for monitoring. 2021-05-04T02:12:27.541Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra session is null 2021-05-04T02:12:27.541Z [main] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:27.994Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=098d2f65-5027-4b63-9967-c87ea048b3bf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:27.995Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=098d2f65-5027-4b63-9967-c87ea048b3bf PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:27.995Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=098d2f65-5027-4b63-9967-c87ea048b3bf ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:28.090Z [main] INFO o.o.s.be.dao.jsongraph.JanusGraphDao RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** JanusGraphDao created 2021-05-04T02:12:30.324Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T02:12:30.324Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T02:12:30.324Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T02:12:30.324Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using native clock to generate timestamps. 2021-05-04T02:12:30.324Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient created 2021-05-04T02:12:30.998Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8ef09d-ed4f-4e80-86de-980a8f7f9e87 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:30.998Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8ef09d-ed4f-4e80-86de-980a8f7f9e87 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:30.998Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8ef09d-ed4f-4e80-86de-980a8f7f9e87 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:33.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3aeff85e-783d-4ad1-a2e1-bbc6b177db74 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:33.994Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3aeff85e-783d-4ad1-a2e1-bbc6b177db74 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:33.994Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=3aeff85e-783d-4ad1-a2e1-bbc6b177db74 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:36.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76bd6ab7-8368-4a31-972a-c425052dd427 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:36.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76bd6ab7-8368-4a31-972a-c425052dd427 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:36.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=76bd6ab7-8368-4a31-972a-c425052dd427 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:39.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68ec87bf-9c7a-4ab7-acce-ac3de1afa499 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:39.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68ec87bf-9c7a-4ab7-acce-ac3de1afa499 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:39.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=68ec87bf-9c7a-4ab7-acce-ac3de1afa499 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:40.141Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:40.141Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:40.141Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:40.149Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:40.203Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:40.203Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:40.203Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:40.207Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:40.226Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:40.226Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:40.226Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:40.229Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:40.305Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:40.305Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:40.305Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:40.308Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:40.324Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:42.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:42.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:42.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:42.138Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:42.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590449b1-17d9-4c2e-8680-b952bd9ffede PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:42.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590449b1-17d9-4c2e-8680-b952bd9ffede PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:42.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=590449b1-17d9-4c2e-8680-b952bd9ffede ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:45.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:45.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:45.114Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:45.155Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:45.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb641937-4232-4a19-92a9-07b19e73725e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:45.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb641937-4232-4a19-92a9-07b19e73725e PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:45.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb641937-4232-4a19-92a9-07b19e73725e ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:48.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:48.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:48.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:48.130Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:48.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2481b585-0ae4-4b16-b92d-f74f37d87275 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:48.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2481b585-0ae4-4b16-b92d-f74f37d87275 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:48.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=2481b585-0ae4-4b16-b92d-f74f37d87275 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:51.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:51.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:51.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:51.137Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:51.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74844650-a3b0-462d-a244-17c1fdb248cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:51.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74844650-a3b0-462d-a244-17c1fdb248cf PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:51.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=74844650-a3b0-462d-a244-17c1fdb248cf ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:54.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:54.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:54.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:54.132Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:54.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ad1ee11-53b1-421b-958d-988c9019aea6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:54.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ad1ee11-53b1-421b-958d-988c9019aea6 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:54.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ad1ee11-53b1-421b-958d-988c9019aea6 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:12:57.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:12:57.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:57.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:12:57.125Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:12:57.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7b574b6-7c10-429c-94dc-8fc3d5ade43b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:12:57.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7b574b6-7c10-429c-94dc-8fc3d5ade43b PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:12:57.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7b574b6-7c10-429c-94dc-8fc3d5ade43b ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:00.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:00.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:00.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:00.129Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:00.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f795c90-e58d-4725-b8af-14b62baab2fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:00.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f795c90-e58d-4725-b8af-14b62baab2fe PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:00.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f795c90-e58d-4725-b8af-14b62baab2fe ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:03.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:03.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:03.114Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:03.145Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:03.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=919dcf2a-1b53-4883-955f-31c53552da49 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:03.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=919dcf2a-1b53-4883-955f-31c53552da49 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:03.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=919dcf2a-1b53-4883-955f-31c53552da49 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:03.998Z [main] INFO c.d.d.c.p.DCAwareRoundRobinPolicy RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using provided data-center name 'Pod' for DCAwareRoundRobinPolicy 2021-05-04T02:13:03.999Z [main] INFO com.datastax.driver.core.Cluster RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.8.171:9042 added 2021-05-04T02:13:03.999Z [main] INFO com.datastax.driver.core.Cluster RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.95.7:9042 added 2021-05-04T02:13:03.999Z [main] INFO com.datastax.driver.core.Cluster RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null New Cassandra host cassandra.onap/10.242.235.132:9042 added 2021-05-04T02:13:04.033Z [cluster2-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.95.7:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:13:04.035Z [cluster2-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.235.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.235.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:13:06.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:06.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:06.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:06.128Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:06.994Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38512e59-45e3-4b66-a57c-f1e8cecd00e4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:06.994Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38512e59-45e3-4b66-a57c-f1e8cecd00e4 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:06.994Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=38512e59-45e3-4b66-a57c-f1e8cecd00e4 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:09.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:09.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:09.114Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:09.128Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:09.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3efa63cb-e0f2-4787-bd94-b7fb1a2ac948 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:09.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3efa63cb-e0f2-4787-bd94-b7fb1a2ac948 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:09.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=3efa63cb-e0f2-4787-bd94-b7fb1a2ac948 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:12.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:12.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:12.116Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:12.133Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:12.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1641730-43df-458c-974f-6fbb82229ae0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:12.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1641730-43df-458c-974f-6fbb82229ae0 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:12.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=a1641730-43df-458c-974f-6fbb82229ae0 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:15.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:15.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:15.123Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:15.216Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:15.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08be7251-52b4-49b9-9910-f44294b1997f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:15.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08be7251-52b4-49b9-9910-f44294b1997f PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:15.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=08be7251-52b4-49b9-9910-f44294b1997f ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:18.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:18.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:18.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:18.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:18.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba98f17c-f9af-4dfd-81f6-968db0811b9e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:18.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba98f17c-f9af-4dfd-81f6-968db0811b9e PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:18.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba98f17c-f9af-4dfd-81f6-968db0811b9e ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:21.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:21.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:21.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:21.130Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:21.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97ea890f-fbe8-4aa0-80fc-e8b010f7fea0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:21.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97ea890f-fbe8-4aa0-80fc-e8b010f7fea0 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:21.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=97ea890f-fbe8-4aa0-80fc-e8b010f7fea0 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:24.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:24.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:24.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:24.203Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:24.541Z [cluster2-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.8.171:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.8.171:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-05-04T02:13:24.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38673ef2-f69f-42e4-b73e-0a64f67b2d6c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:24.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38673ef2-f69f-42e4-b73e-0a64f67b2d6c PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:24.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=38673ef2-f69f-42e4-b73e-0a64f67b2d6c ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:27.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:27.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:27.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d82a9497-8569-4720-9c95-b1412e88eeb5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:27.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d82a9497-8569-4720-9c95-b1412e88eeb5 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:27.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=d82a9497-8569-4720-9c95-b1412e88eeb5 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:29.574Z [main] INFO o.o.s.b.d.c.AuditCassandraDao RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** AuditCassandraDao failed 2021-05-04T02:13:29.595Z [main] WARN o.s.w.c.s.XmlWebApplicationContext RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED 2021-05-04T02:13:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:30.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:30.127Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:30.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feffbaa3-1a6d-47d3-8fd9-90ca50258812 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:13:30.993Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feffbaa3-1a6d-47d3-8fd9-90ca50258812 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:13:30.993Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=feffbaa3-1a6d-47d3-8fd9-90ca50258812 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:13:31.605Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T02:13:31.606Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T02:13:31.606Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** sdcSchemaUtils cluster closed 2021-05-04T02:13:31.608Z [main] WARN o.s.c.a.CommonAnnotationBeanPostProcessor RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Destroy method on bean with name 'janusgraph-client' threw an exception: java.lang.NullPointerException 2021-05-04T02:13:31.618Z [main] ERROR o.s.web.context.ContextLoader RequestID=a971f5d5-a112-4ab6-9d5f-fb4ace3fe227 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Context initialization failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 59 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 73 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-14644900250238799573.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 87 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:415) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 101 common frames omitted Caused by: java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.openecomp.sdc.be.dao.cassandra.AuditCassandraDao.init(AuditCassandraDao.java:65) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ... 114 common frames omitted 2021-05-04T02:13:33.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:33.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:33.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:33.124Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:36.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:36.125Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:39.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:39.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:39.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:39.124Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:42.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:42.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:42.114Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:13:42.207Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:45.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:13:45.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:13:45.122Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:14:01.219Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:14:01.219Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:14:01.219Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:14:01.243Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:14:27.161Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:14:27.161Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:14:27.161Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:14:27.169Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:14:45.969Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:14:45.969Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:14:45.970Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:14:45.972Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:14:59.585Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:14:59.585Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:14:59.585Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:14:59.591Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:15:11.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:15:11.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:15:11.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:15:11.705Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:15:26.924Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:15:26.925Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:15:26.925Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:15:26.931Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.359Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.359Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.359Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.366Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.405Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.406Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.406Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.408Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.420Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.420Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.420Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.422Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.503Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.503Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.503Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.506Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.519Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.519Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.519Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.521Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.603Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.603Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.603Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.606Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.619Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.619Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.619Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.621Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.638Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.638Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.638Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.640Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.704Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.704Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.704Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.706Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.724Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.724Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.724Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.726Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.802Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.802Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.802Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.804Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.816Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.816Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.816Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.818Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.833Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.834Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.834Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.835Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.903Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.903Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.903Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.905Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:01.929Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.929Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:01.929Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:01.931Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.007Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.007Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.007Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.009Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.022Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.022Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.022Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.024Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.103Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.103Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.103Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.105Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.133Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.134Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.134Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.136Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.206Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.206Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.206Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.208Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.221Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.221Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.221Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.223Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.303Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.303Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.303Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.305Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.323Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.323Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.323Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.325Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.403Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.403Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.403Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.405Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.420Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.420Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.420Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.423Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.503Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.503Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.503Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.505Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.518Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.519Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.519Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.520Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.548Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.548Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.549Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.550Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.605Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.605Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.605Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.609Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.712Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.713Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.713Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.715Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.731Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.731Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.731Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.733Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.803Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.803Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.803Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.805Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.816Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.816Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.816Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.817Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.829Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.830Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.830Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.831Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.909Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.910Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.910Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.912Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.925Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.925Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.925Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.927Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:02.942Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.942Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:02.942Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:02.944Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:03.002Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:03.002Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:03.002Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:03.004Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:03.018Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:03.018Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:03.018Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:03.020Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:03.030Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:03.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:03.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:03.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:03.124Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:06.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:06.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:06.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:06.122Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:09.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:09.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:09.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:09.121Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:12.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:12.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:12.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:12.123Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:15.117Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:15.118Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:15.122Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:15.136Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:18.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:18.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:18.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:18.125Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:21.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:21.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:21.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:21.124Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:24.113Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:24.113Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:24.115Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:24.124Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:27.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:27.123Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:30.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:30.123Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:33.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:33.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:33.129Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:33.142Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:36.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:36.127Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:39.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:39.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:39.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:39.123Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:42.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:42.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:42.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:42.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:45.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:45.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:45.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:45.124Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:48.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:48.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:48.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:48.125Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:51.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:51.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:51.111Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:51.120Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:54.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:54.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:54.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:54.125Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:57.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:16:57.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:16:57.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:16:57.124Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:00.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:00.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:00.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:00.123Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:03.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:03.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:03.114Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:03.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:06.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:06.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:06.116Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:06.129Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:09.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:09.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:09.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:09.122Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:12.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:12.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:12.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:12.121Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:15.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:15.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:15.114Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:15.129Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:18.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:18.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:18.114Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:18.130Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:21.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:21.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:21.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:21.122Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:24.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:24.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:24.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:24.135Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:27.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:27.122Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:30.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:30.122Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:33.111Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:33.112Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:33.115Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:33.129Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:36.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:36.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:39.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:39.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:39.117Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:17:58.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:58.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:17:58.127Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:17:58.129Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.739Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.739Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.739Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.745Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.773Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.773Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.773Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.774Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.785Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.785Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.785Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.787Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.795Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.795Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.795Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.796Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.804Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.804Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.804Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.806Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.820Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.820Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.820Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.822Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.832Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.832Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.832Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.834Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.849Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.849Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.849Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.851Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.864Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.864Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.864Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.866Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.882Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.882Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:14.882Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:14.884Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:14.896Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:15.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:15.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:15.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:15.133Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:18.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:18.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:18.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:18.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:21.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:21.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:21.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:21.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:24.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:24.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:24.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:24.125Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:27.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:27.133Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:30.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:30.202Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:33.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:33.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:33.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:33.123Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:36.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:36.129Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:39.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:39.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:39.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:39.122Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:42.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:42.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:42.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:42.122Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:45.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:45.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:45.115Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:18:57.225Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:57.226Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:18:57.226Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:18:57.229Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:09.343Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:09.343Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:09.343Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:09.347Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.460Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.460Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.460Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.467Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.480Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.480Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.480Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.483Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.492Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.492Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.492Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.494Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.505Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.505Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.505Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.506Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.524Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.524Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.524Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.526Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.540Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.540Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.540Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.542Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.552Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.552Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.552Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.554Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.564Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.564Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.564Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.566Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.575Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.575Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.575Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.577Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.587Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.587Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:21.587Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:21.588Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:21.598Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:24.112Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:24.112Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:24.117Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:24.129Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:27.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:27.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:27.122Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:30.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:30.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:30.123Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:33.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:33.112Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:33.115Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:33.126Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:19:36.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:19:36.112Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:19:36.123Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:00.554Z [main] WARN o.o.s.c.util.YamlToObjectConverter RequestID=null ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= The file /var/lib/jetty/config/catalog-be/neo4j-errors-configuration.yaml cannot be found. Ignore reading configuration. 2021-05-04T02:20:00.714Z [main] INFO o.o.s.be.listen.BEAppContextListener RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ASDC version from MANIFEST is 1.8.5 2021-05-04T02:20:00.770Z [main] INFO o.o.s.b.m.BeMonitoringService RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= BE monitoring service is disabled 2021-05-04T02:20:00.770Z [main] INFO o.s.web.context.ContextLoader RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Root WebApplicationContext: initialization started 2021-05-04T02:20:02.958Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphClient created 2021-05-04T02:20:02.962Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** createGraph started ** 2021-05-04T02:20:02.962Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:02.962Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:03.361Z [main] INFO com.datastax.driver.core RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= DataStax Java driver 3.8.0 for Apache Cassandra 2021-05-04T02:20:03.366Z [main] INFO c.d.driver.core.GuavaCompatibility RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Detected Guava >= 19 in the classpath, using modern compatibility layer 2021-05-04T02:20:03.968Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T02:20:04.154Z [main] INFO com.datastax.driver.core.NettyUtil RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Found Netty's native epoll transport in the classpath, using it 2021-05-04T02:20:04.663Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:04.664Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Scheduling reconnect task org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask@22c14d10 with interval of 3 seconds 2021-05-04T02:20:04.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:04.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:04.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:04.703Z [main] INFO o.o.s.b.d.j.JanusGraphGenericDao RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphGenericDao created 2021-05-04T02:20:04.718Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:04.982Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= All sdc keyspaces are : [sdcartifact, sdcrepository, sdcaudit] 2021-05-04T02:20:04.984Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connecting to node: [cassandra.onap] port: 9042. 2021-05-04T02:20:04.985Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connection timeout in seconds : 120 2021-05-04T02:20:04.985Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T02:20:04.985Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T02:20:04.985Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T02:20:04.986Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T02:20:04.986Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** cluster created 2021-05-04T02:20:04.986Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= creating cluster for Cassandra Health Check. 2021-05-04T02:20:05.012Z [main] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null create cassandra cluster failed with exception.: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:20:05.044Z [main] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ECOMP Portal health check configuration is missing. 2021-05-04T02:20:05.061Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Enter init method of SwitchoverDetector 2021-05-04T02:20:05.062Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null switchover detector service is disabled 2021-05-04T02:20:05.068Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null creating cluster for Cassandra for monitoring. 2021-05-04T02:20:05.068Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra session is null 2021-05-04T02:20:05.068Z [main] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Failed to connect to cassandra 2021-05-04T02:20:06.048Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96da6f74-b483-43b6-ad5c-c567a162ebd3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:20:06.049Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96da6f74-b483-43b6-ad5c-c567a162ebd3 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:20:06.049Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=96da6f74-b483-43b6-ad5c-c567a162ebd3 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:20:06.166Z [main] INFO o.o.s.be.dao.jsongraph.JanusGraphDao RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** JanusGraphDao created 2021-05-04T02:20:07.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:07.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:07.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:07.707Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:08.232Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T02:20:08.232Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T02:20:08.232Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T02:20:08.232Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using native clock to generate timestamps. 2021-05-04T02:20:08.233Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient created 2021-05-04T02:20:08.254Z [main] INFO o.o.s.b.d.c.AuditCassandraDao RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** AuditCassandraDao failed 2021-05-04T02:20:08.263Z [main] WARN o.s.w.c.s.XmlWebApplicationContext RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED 2021-05-04T02:20:09.047Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e43c2b33-2942-4e95-bd60-3eeecf5508a9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:20:09.047Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e43c2b33-2942-4e95-bd60-3eeecf5508a9 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:20:09.048Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=e43c2b33-2942-4e95-bd60-3eeecf5508a9 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:20:10.265Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T02:20:10.267Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T02:20:10.267Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** sdcSchemaUtils cluster closed 2021-05-04T02:20:10.269Z [main] WARN o.s.c.a.CommonAnnotationBeanPostProcessor RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Destroy method on bean with name 'janusgraph-client' threw an exception: java.lang.NullPointerException 2021-05-04T02:20:10.282Z [main] ERROR o.s.web.context.ContextLoader RequestID=b9d4ce96-1313-47cb-ab85-962898975f2f ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Context initialization failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 59 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 73 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-8185949618209926720.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 87 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:415) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 101 common frames omitted Caused by: java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.openecomp.sdc.be.dao.cassandra.AuditCassandraDao.init(AuditCassandraDao.java:65) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ... 114 common frames omitted 2021-05-04T02:20:10.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:10.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:10.669Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:10.766Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:13.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:13.690Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:16.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:16.684Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:19.671Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:19.748Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:22.669Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:22.688Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:25.670Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:25.691Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:28.672Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:28.701Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:31.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:31.680Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:34.670Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:34.703Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:37.669Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:37.684Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:40.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:40.685Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:43.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:43.704Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:46.670Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:46.696Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:49.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:49.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:49.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:49.684Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:52.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:52.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:52.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:52.706Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:55.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:55.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:55.680Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:55.703Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:58.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:20:58.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:20:58.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:20:58.680Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:01.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:01.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:01.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:01.703Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:04.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:04.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:04.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:04.707Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:07.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:07.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:07.671Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:07.689Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:10.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:10.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:10.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:10.685Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:13.669Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:13.708Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:16.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:16.684Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:19.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:19.681Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:22.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:22.706Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:25.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:25.681Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:28.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:28.681Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:31.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:31.683Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:34.666Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:34.705Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:37.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:37.683Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:40.669Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:21:52.893Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:52.893Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:21:52.893Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:21:52.899Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:05.013Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:05.014Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:05.014Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:05.017Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.134Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.134Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.134Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.140Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.154Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.154Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.154Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.156Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.201Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.202Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.202Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.204Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.216Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.217Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.217Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.219Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.231Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.231Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.231Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.233Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.302Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.302Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.302Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.304Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.318Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.318Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.318Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.320Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.333Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.333Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.333Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.335Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.401Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.402Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.402Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.404Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.416Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.416Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:17.417Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:17.419Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:17.433Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:19.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:19.703Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:22.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:22.755Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:25.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:25.683Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:28.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:28.701Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:31.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:31.680Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:34.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:34.679Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:37.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:37.704Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:40.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:40.711Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:43.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:43.681Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:46.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:22:46.680Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:49.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:22:49.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:22:49.670Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:23:01.784Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:23:01.784Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:23:01.784Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:23:01.788Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:23:13.907Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:23:13.908Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:23:13.908Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:23:13.912Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:23:33.234Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:23:33.235Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:23:33.235Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:23:33.241Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:23:57.653Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:23:57.653Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:23:57.653Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:23:57.656Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:10.566Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.566Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:10.566Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.572Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:10.607Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.607Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:10.607Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.610Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:10.700Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.700Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:10.700Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.702Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:10.719Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.719Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:10.719Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.721Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:10.801Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.802Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:10.802Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.804Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:10.819Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.819Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:10.819Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.821Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:10.899Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.900Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:10.900Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.902Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:10.919Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.919Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:10.919Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.922Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:10.938Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.939Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:10.939Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:10.942Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.001Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.001Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.001Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.004Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.019Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.019Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.019Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.021Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.110Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.113Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.202Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.202Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.202Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.204Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.218Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.218Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.218Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.220Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.308Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.308Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.308Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.309Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.325Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.325Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.325Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.328Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.341Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.341Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.341Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.344Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.358Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.358Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.358Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.360Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.373Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.373Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.373Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.375Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.390Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.390Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.390Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.391Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.401Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.401Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.401Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.404Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.414Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.414Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.414Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.416Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.431Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.431Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:11.431Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:11.433Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:11.443Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:13.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:13.681Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:16.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:16.683Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:19.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:19.701Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:22.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:22.691Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:25.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:25.681Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:28.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:28.701Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:31.666Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:31.670Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:31.685Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:34.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:34.683Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:37.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:37.707Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:40.672Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:40.688Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:43.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:43.704Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:46.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:46.681Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:49.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:49.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:49.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:49.680Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:52.666Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:52.666Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:52.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:52.678Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:55.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:55.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:55.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:55.703Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:58.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:24:58.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:24:58.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:24:58.678Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:01.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:01.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:01.666Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:01.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:04.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:04.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:04.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:04.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:07.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:07.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:07.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:07.680Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:10.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:10.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:10.668Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:10.684Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:13.669Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:13.680Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:16.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:16.680Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:19.666Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:19.701Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:22.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:22.679Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:25.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:25.679Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:28.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:28.700Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:31.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:31.678Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:34.674Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:34.690Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:37.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:37.701Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:40.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:40.700Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:43.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:43.679Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:46.670Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:25:58.780Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:58.780Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:25:58.780Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:25:58.783Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:15.398Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:15.398Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:15.398Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:15.404Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.517Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.517Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.517Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.523Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.539Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.539Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.539Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.541Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.552Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.552Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.552Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.553Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.567Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.567Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.567Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.569Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.587Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.587Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.587Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.590Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.608Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.608Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.608Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.611Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.624Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.624Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.624Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.625Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.636Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.636Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.636Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.638Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.655Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.655Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.655Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.657Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.667Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.667Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.667Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.669Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.694Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.694Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:27.694Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:27.696Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:27.705Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:28.666Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:28.675Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:31.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:31.679Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:34.669Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:34.705Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:37.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:37.678Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:40.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:40.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:40.686Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:43.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:43.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:43.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:46.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:46.666Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:46.675Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:49.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:49.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:49.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:49.678Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:52.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:52.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:52.666Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:52.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:55.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:55.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:55.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:55.705Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:58.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:26:58.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:26:58.669Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:26:58.714Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:01.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:01.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:01.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:01.677Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:04.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:04.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:04.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:04.679Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:07.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:07.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:07.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:07.703Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:10.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:10.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:10.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:10.700Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:13.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:13.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:13.680Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:16.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:16.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:16.678Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:19.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:19.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:19.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:22.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:22.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:22.679Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:25.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:25.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:25.679Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:28.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:28.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:28.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:31.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:31.670Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:31.681Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:34.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:34.666Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:34.702Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:27:37.665Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:27:37.667Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:27:37.679Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:00.638Z [main] WARN o.o.s.c.util.YamlToObjectConverter RequestID=null ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= The file /var/lib/jetty/config/catalog-be/neo4j-errors-configuration.yaml cannot be found. Ignore reading configuration. 2021-05-04T02:28:00.784Z [main] INFO o.o.s.be.listen.BEAppContextListener RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ASDC version from MANIFEST is 1.8.5 2021-05-04T02:28:00.831Z [main] INFO o.o.s.b.m.BeMonitoringService RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= BE monitoring service is disabled 2021-05-04T02:28:00.832Z [main] INFO o.s.web.context.ContextLoader RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Root WebApplicationContext: initialization started 2021-05-04T02:28:02.733Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphClient created 2021-05-04T02:28:02.736Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** createGraph started ** 2021-05-04T02:28:02.736Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:02.736Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:03.072Z [main] INFO com.datastax.driver.core RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= DataStax Java driver 3.8.0 for Apache Cassandra 2021-05-04T02:28:03.076Z [main] INFO c.d.driver.core.GuavaCompatibility RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Detected Guava >= 19 in the classpath, using modern compatibility layer 2021-05-04T02:28:03.690Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T02:28:03.872Z [main] INFO com.datastax.driver.core.NettyUtil RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Found Netty's native epoll transport in the classpath, using it 2021-05-04T02:28:04.326Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:04.327Z [main] INFO o.o.s.b.d.j.JanusGraphClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Scheduling reconnect task org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$ReconnectTask@2e778abb with interval of 3 seconds 2021-05-04T02:28:04.338Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:04.339Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:04.343Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:04.371Z [main] INFO o.o.s.b.d.j.JanusGraphGenericDao RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** JanusGraphGenericDao created 2021-05-04T02:28:04.409Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:04.702Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= All sdc keyspaces are : [sdcartifact, sdcrepository, sdcaudit] 2021-05-04T02:28:04.702Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connecting to node: [cassandra.onap] port: 9042. 2021-05-04T02:28:04.703Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Connection timeout in seconds : 120 2021-05-04T02:28:04.703Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T02:28:04.703Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T02:28:04.703Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T02:28:04.703Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= Using native clock to generate timestamps. 2021-05-04T02:28:04.704Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= ** cluster created 2021-05-04T02:28:04.704Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=WARN ServerIPAddress=10.242.235.154 ServiceName= TargetServiceName=null ErrorCode=900 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity= creating cluster for Cassandra Health Check. 2021-05-04T02:28:07.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:07.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:07.332Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:07.352Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:10.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:10.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:10.332Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:10.403Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:13.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:13.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:13.336Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:13.354Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:16.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:16.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:16.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:16.357Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:19.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:19.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:19.332Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:19.350Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:22.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:22.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:22.332Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:22.349Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:25.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:25.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:25.333Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:25.351Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:28.305Z [main] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null create cassandra cluster failed with exception.: Authentication error on host cassandra.onap/10.242.95.7:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-05-04T02:28:28.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:28.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:28.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:28.344Z [main] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ECOMP Portal health check configuration is missing. 2021-05-04T02:28:28.371Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Enter init method of SwitchoverDetector 2021-05-04T02:28:28.372Z [main] INFO o.o.s.b.s.d.SwitchoverDetector RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null switchover detector service is disabled 2021-05-04T02:28:28.379Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null creating cluster for Cassandra for monitoring. 2021-05-04T02:28:28.379Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null The cassandra session is null 2021-05-04T02:28:28.380Z [main] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:28.402Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:28.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1ce23243-d4cd-442e-a3ac-f2d6ae4afb86 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:28:28.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1ce23243-d4cd-442e-a3ac-f2d6ae4afb86 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:28:28.964Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=1ce23243-d4cd-442e-a3ac-f2d6ae4afb86 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:29.075Z [main] INFO o.o.s.be.dao.jsongraph.JanusGraphDao RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** JanusGraphDao created 2021-05-04T02:28:31.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:31.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:31.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:31.370Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:31.448Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketConnectTimeout was provided, setting Cassandra client to use SocketConnectTimeout: 20000 . 2021-05-04T02:28:31.448Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null SocketReadTimeout was provided, setting Cassandra client to use SocketReadTimeout: 20000 . 2021-05-04T02:28:31.448Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null localDatacenter was provided, setting Cassndra clint to use datacenter: Pod as local. 2021-05-04T02:28:31.448Z [main] INFO c.datastax.driver.core.ClockFactory RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Using native clock to generate timestamps. 2021-05-04T02:28:31.448Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient created 2021-05-04T02:28:31.955Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f313a786-3f07-44dc-bf17-f314951acba1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:28:31.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f313a786-3f07-44dc-bf17-f314951acba1 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:28:31.956Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=f313a786-3f07-44dc-bf17-f314951acba1 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:34.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:34.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:34.335Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:34.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8158715-51c1-45f5-8911-76dd015ae2d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:28:34.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8158715-51c1-45f5-8911-76dd015ae2d8 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:28:34.956Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8158715-51c1-45f5-8911-76dd015ae2d8 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:37.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86fefa48-d57a-4320-9e10-72e59cbe848c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:28:37.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86fefa48-d57a-4320-9e10-72e59cbe848c PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:28:37.956Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=86fefa48-d57a-4320-9e10-72e59cbe848c ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:40.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c78cdc3f-ab1a-4178-8ea8-aef9c81ccbd6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:28:40.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c78cdc3f-ab1a-4178-8ea8-aef9c81ccbd6 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:28:40.956Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=c78cdc3f-ab1a-4178-8ea8-aef9c81ccbd6 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:43.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3521832-e900-4ac6-a070-d3a916d2130e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:28:43.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3521832-e900-4ac6-a070-d3a916d2130e PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:28:43.956Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3521832-e900-4ac6-a070-d3a916d2130e ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:46.452Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:46.452Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:28:46.452Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:28:46.455Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:28:46.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ceba6343-ccb5-428b-b992-c516dee468e8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:28:46.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ceba6343-ccb5-428b-b992-c516dee468e8 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:28:46.956Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=ceba6343-ccb5-428b-b992-c516dee468e8 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:49.955Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55193918-11c0-4ac8-94d3-5a9cc8988a57 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:28:49.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55193918-11c0-4ac8-94d3-5a9cc8988a57 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:28:49.956Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=55193918-11c0-4ac8-94d3-5a9cc8988a57 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:51.585Z [main] INFO o.o.s.b.d.c.AuditCassandraDao RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** AuditCassandraDao failed 2021-05-04T02:28:51.603Z [main] WARN o.s.w.c.s.XmlWebApplicationContext RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED 2021-05-04T02:28:52.955Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a00d20a-1647-43e7-8104-fb2650739058 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-05-04T02:28:52.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a00d20a-1647-43e7-8104-fb2650739058 PartnerName=SDC.BE The cassandra session is null 2021-05-04T02:28:52.956Z [BE-Health-Check-Task] ERROR o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a00d20a-1647-43e7-8104-fb2650739058 ErrorCategory=ERROR ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null Failed to connect to cassandra 2021-05-04T02:28:53.592Z [main] INFO o.o.s.b.d.cassandra.CassandraClient RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T02:28:53.594Z [main] INFO o.o.s.b.d.c.schema.SdcSchemaUtils RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** CassandraClient cluster closed 2021-05-04T02:28:53.594Z [main] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null ** sdcSchemaUtils cluster closed 2021-05-04T02:28:53.596Z [main] WARN o.s.c.a.CommonAnnotationBeanPostProcessor RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Destroy method on bean with name 'janusgraph-client' threw an exception: java.lang.NullPointerException 2021-05-04T02:28:53.610Z [main] ERROR o.s.web.context.ContextLoader RequestID=05426dfc-4e2b-4a38-be6c-c506295546b3 ErrorCategory=ERROR ServerIPAddress=10.242.235.154 ServiceName=SDC catalog TargetServiceName=null ErrorCode=500 ServerFQDN=dev-sdc-be-69f59769db-d8vzh TargetEntity=null Context initialization failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'additionalInformationServlet' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/servlets/AdditionalInformationServlet.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userBusinessLogic' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/user/UserBusinessLogic.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 59 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'componentUtils' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/impl/ComponentsUtils.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 73 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditingManager' defined in file [/tmp/jetty-0_0_0_0-8443-catalog-be_war-_-any-7026841069390689362.dir/webapp/WEB-INF/classes/org/openecomp/sdc/be/auditing/impl/AuditingManager.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 87 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'audit-cassandra-dao': Invocation of init method failed; nested exception is java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:415) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ... 101 common frames omitted Caused by: java.lang.RuntimeException: Audit keyspace [sdcaudit] failed to connect with error : KEYSPACE_NOT_CONNECTED at org.openecomp.sdc.be.dao.cassandra.AuditCassandraDao.init(AuditCassandraDao.java:65) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ... 114 common frames omitted 2021-05-04T02:29:08.776Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.776Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.776Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.786Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.804Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.805Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.805Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.807Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.822Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.822Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.822Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.825Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.838Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.838Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.838Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.841Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.854Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.855Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.855Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.857Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.872Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.872Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.872Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.874Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.888Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.888Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.888Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.890Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.903Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.903Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.903Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.905Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.922Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.922Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.922Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.924Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.936Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.936Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:08.936Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:08.938Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:08.951Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:10.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:10.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:10.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:10.342Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:13.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:13.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:13.334Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:13.346Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:16.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:16.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:16.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:16.405Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:19.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:19.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:19.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:19.345Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:22.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:22.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:22.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:22.405Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:25.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:25.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:25.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:25.404Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:28.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:28.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:28.329Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:28.340Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:31.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:31.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:31.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:31.347Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:34.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:34.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:34.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:34.403Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:37.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:37.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:37.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:37.344Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:40.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:40.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:40.332Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:40.403Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:43.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:43.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:43.329Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:43.359Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:46.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:46.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:46.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:46.347Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:49.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:49.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:49.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:49.401Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:52.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:52.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:52.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:52.347Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:55.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:55.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:55.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:55.346Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:58.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:29:58.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:29:58.329Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:29:58.402Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:01.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:01.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:01.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:01.344Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:04.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:04.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:04.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:04.346Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:07.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:07.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:07.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:07.344Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:10.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:10.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:10.329Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:10.400Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:13.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:13.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:13.333Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:13.349Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:16.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:16.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:16.329Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:16.345Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:19.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:19.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:19.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:19.401Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:22.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:22.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:22.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:22.348Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:25.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:25.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:25.332Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:25.346Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:28.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:28.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:28.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:28.402Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:31.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:31.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:31.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:31.352Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:34.330Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:34.330Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:34.333Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:34.347Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:37.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:37.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:37.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:37.401Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:40.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:40.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:40.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:40.344Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:43.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:43.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:43.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:43.346Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:46.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:46.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:46.335Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:46.350Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:49.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:49.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:49.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:49.343Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:52.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:52.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:52.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:52.348Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:55.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:55.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:55.331Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:55.352Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:58.327Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient ** open graph with /var/lib/jetty/config/catalog-be/janusgraph.properties started 2021-05-04T02:30:58.328Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient openGraph : try to load file /var/lib/jetty/config/catalog-be/janusgraph.properties 2021-05-04T02:30:58.330Z [JanusGraph-Reconnect-Task] INFO c.datastax.driver.core.ClockFactory Using native clock to generate timestamps. 2021-05-04T02:30:58.350Z [JanusGraph-Reconnect-Task] INFO o.o.s.b.d.j.JanusGraphClient createGraph : failed to open JanusGraph graph with configuration file: /var/lib/jetty/config/catalog-be/janusgraph.properties